INCREMENTAL AGGREGATION IN INFORMATICA

Theory

Incremental Aggregation is the process of capturing the changes in the source and calculating the aggregations in a session. This process makes the integration service to update the target incrementally and avoids the process of calculating the aggregations on the entire source. Consider the below sales table as an example and see how the incremental aggregation works.

Source:

YEAR PRICE
----------
2010 100
2010 200
2010 300
2011 500
2011 600
2012 700

For simplicity, I have used only the year and price columns of sales table. We need to do aggregation and find the total price in each year.

When you run the session for the first time using the incremental aggregation, then integration service process the entire source and stores the data in two file, index and data file. The integration service creates the files in the cache directory specified in the aggregator transformation properties.

After the aggregation, the target table will have the below data.

Target:

YEAR PRICE
----------
2010 600
2011 1100
2012 700

Now assume that the next day few more rows are added into the source table.

Source:

YEAR PRICE

----------

2010 100

2010 200

2010 300

2011 500

2011 600

2012 700

2010 400

2011 100

2012 200

2013 800

Now for the second run, you have to pass only the new data changes to the incremental aggregation. So, the source will contain the last four records. The incremental aggregation uses the data stored in the cache and calculates the aggregation. Once the aggregation is done, the integration service writes the changes to the target and the cache. The target table will contains the below data.

Target:

YEAR PRICE
----------
2010 1000
2011 1200
2012 900
2013 800

Points to remember

• When you use incremental aggregation, first time you have to run the session with complete source data and in the subsequent runs you have to pass only the changes in the source data. 
• Use incremental aggregation only if the target is not going to change significantly. If the incremental aggregation process changes more than half of the data in target, then the session performance many not benefit. In this case go for normal aggregation.

Note : The integration service creates a new aggregate cache when

1. A new version of mapping is saved 
2. Configure the session to reinitialize the aggregate cache 
3. Moving or deleting the aggregate files 
4. Decreasing the number of partitions

Configuring the mapping for incremental aggregation

Before enabling the incremental aggregation option, make sure that you capture the changes in the source data. You can use lookup transformation or stored procedure transformation to remove the data which is already processed. You can also create a trigger on the source database and can read only the source changes in the mapping.

You can find the incremental aggregation in the session properties tab under the performance section.

Performing Update without Look-Up and Update strategy.

Hi Folks,

This sounds a bit awkward that, when we have Lookup and Update strategy, why not to use it.

But trust me this is a very commonly encountered question in interviews. So here I can give you a solution for this with descriptions.

Please go through it twice, you will definitely ably to make it out.

You might have come across an ETL scenario, where you need to update a huge table with few records and occasional inserts. The straight forward approach of using LookUp transformation to identify the Inserts, Update and Update Strategy to do the Insert or Update may not be right for this particular scenario, mainly because of the LookUp transformation may not perform better and start degrading as the lookup table size increases.
In this article lets talk about a design, which can take care of the scenario we just spoke.

The Theory

When you configure an Informatica PowerCenter session, you have several options for handling database operations such as insert, update, delete.

Specifying an Operation for All Rows

During session configuration, you can select a single database operation for all rows using the Treat Source Rows As setting from the 'Properties' tab of the session.

• Insert :- Treat all rows as inserts. 
• Delete :- Treat all rows as deletes. 
• Update :- Treat all rows as updates. 
• Data Driven :- Integration Service follows instructions coded into Update Strategy flag rows for insert, delete, update, or reject.

Specifying Operations for Individual Target Rows

Once you determine how to treat all rows in the session, you can also set options for individual rows, which gives additional control over how each rows behaves. Define these options in the Transformations view on Mapping tab of the session properties. 

 
Insert :- Select this option to insert a row into a target table.
Delete :- Select this option to delete a row from a table.
Update :- You have the following options in this situation:

• Update as Update :- Update each row flagged for update if it exists in the target table. 
• Update as Insert :- Insert each row flagged for update. 
• Update else Insert :- Update the row if it exists. Otherwise, insert it. 

Truncate Table :- Select this option to truncate the target table before loading data.

Design and Implementation

Now we understand the properties we need to use for our design implementation.

We can create the mapping just like an 'INSERT' only mapping, with out LookUp, Update Strategy Transformation. During the session configuration lets set up the session properties such that the session will have the capability to both insert and update.

First set Treat Source Rows As property as shown in below image.

            


Now lets set the properties for the target table as shown below. Choose the properties Insert and Update else Insert.

That's all we need to set up the session for update and insert with out update strategy.

All The Best

Getting ALIVE..!!

Hi All

I was not active on my blogs from a long time. As I was tangled in many things. Now here again I am back to bring some new things which might help you guys in one or the other way.

My initial blog were on .NET, then I moved to Mainframes, as I was hired as a Mainframe professional. Now I am working in an MNC as an Informatica and cognos developer, so here I want to share some of the important things.

Hope this will help you all.

Thanks

Some common DB2 / SQL errors with their resolution.

Hi

I am here presenting some common errors, hope they will help you in interviews.

INSTANT EXPERT: SQL ERROR CODES

DB2 UDB Version 8 for z/OS

Introduction

SQL return codes provided by DB2 UDB for OS/390 and z/OS can be confusing and often reference manuals are not available or close at hand when you really need 

them. This Instant Expert Reference Card will review SQL 

return code processing and common SQL error conditions you may encounter in your daily work with DB2.

Retrieving SQL Return Code Information & 

Messages Into Your Programs

• COBOL programs executing SQL statements communicate 

with DB2 via a Working Storage area called the SQL Communications Area (SQLCA).

• When DB2 executes SQL statements, it returns the results of 

the operation into the SQLCODE and SQLSTATE fields in the 

SQLCA. SQLCODE provides key information about the success or failure of SQL statement execution.

SQLCODE Overview

• If SQLCODE = 0, execution was successful.

• If SQLCODE > 0, execution was successful with a warning.

• If SQLCODE < 0, execution was not successful.

• -805 DBRM OR PACKAGE NAME location-name.collectionid.dbrm-name.consistency -token NOT FOUND IN PLAN 

plan-name. REASON reason. 

Suggestion: Ensure COLLECTION 

name is in DB2 PLAN. Recompile and BIND the DB2 program. 

Verify correct LOAD library is being used.

• -811 THE RESULT OF AN EMBEDDED SELECT STATEMENT OR 

A SUBSELECT IN THE SET CLAUSE OF AN UPDATE STATEMENT IS A TABLE OF MORE THAN ONE ROW, OR THE RESULT 

OF A SUBQUERY OF A BASIC PREDICATE IS MORE THAN ONE 

VALUE.

 Suggestion: -811 is often detected after program 

check for DB2 data existence. Consider using new DB2 V8 

FETCH FIRST ROW ONLY feature instead.

• -818 THE PRECOMPILER-GENERATED TIMESTAMP x IN THE 

LOAD MODULE IS DIFFERENT FROM THE BIND TIMESTAMP y 

BUILT FROM THE DBRM z.

Suggestion: Recompile and BIND 

the DB2 program. Verify correct LOAD library is being used.

• -904 UNSUCCESSFUL EXECUTION CAUSED BY AN UNAVAILABLE RESOURCE. REASON reason-code, TYPE OF RESOURCE 

resource-type, AND RESOURCE NAME resource-name.

 Suggestion: -904 is usually caused because a database utility job has 

started the desired DB2 object in utility mode. Check DB2 Master Log for more details on the resource name – contact DBA. 

• -911 THE CURRENT UNIT OF WORK HAS BEEN ROLLED BACK 

DUE TO DEADLOCK OR TIMEOUT. REASON reason-code, 

TYPE OF RESOURCE resource-type, AND RESOURCE NAME 

resource-name.

Suggestion: Review DB2 Master Log to find 

process holding DB2 locks. Consider adding additional COMMITs to program holding the DB2 resource. 

• -913 UNSUCCESSFUL EXECUTION CAUSED BY DEADLOCK OR 

TIMEOUT. REASON CODE reason-code, TYPE OF RESOURCE 

resource-type, AND RESOURCE NAME resource-name.

Suggestion: Review DB2 Master Log to find process holding DB2

locks. Consider adding additional COMMITs to program holding the DB2 resource. 

• -922 AUTHORIZATION FAILURE: error-type ERROR. REASON 

reason-code.

 Suggestion: Connection to DB2 has failed 

due authority for USER or PLAN. Contact DBA to check DB2 

authorizations.

• -927 THE LANGUAGE INTERFACE (LI) WAS CALLED WHEN THE 

CONNECTING ENVIRONMENT WAS NOT ESTABLISHED. THE 

PROGRAM SHOULD BE INVOKED UNDER THE DSN COMMAND.

·        -117 The number of values assigned is not the same as 

the number of specified or implied columns. 

Suggestion: Provide one value for each column in the table.

• -150 (DB2 V8) the object of the insert, delete, or update 

statement is a view, system-maintained materialized 

query table, or transition table for which the requested operation is not permitted. Suggestion: Be certain to 

specify base DB2 table/view names for INSERT statements.

• -180 THE DATE, TIME, OR TIMESTAMP VALUE value IS INVALID. 

Suggestion: Verify the data value is in the correct range and 

value type.

• -181 THE STRING REPRESENTATION OF A DATETIME VALUE IS 

NOT A VALID DATETIME VALUE.

Suggestion: Verify data format 

with the SQL Reference Guide.

• -204 name IS AN UNDEFINED NAME.

Suggestion: Correct DB2

CREATOR or OBJECT NAMEs located in SQL statements.

• -227 (DB2 V8) FETCH fetch-orientation IS NOT ALLOWED, 

BECAUSE CURSOR cursor-name HAS AN UNKNOWN POSITION 

(sqlcode,sqlstate).

 Suggestion: CLOSE and re-OPEN the cursor; 

For scrollable use (FIRST, LAST, BEFORE, AFTER, or ABSOLUTE) 

to establish valid position.

• -305 THE NULL VALUE CANNOT BE ASSIGNED TO OUTPUT HOST 

VARIABLE NUMBER position-number BECAUSE NO INDICATOR 

VARIABLE IS SPECIFIED.

 Suggestion: Add null indicator variable 

to SELECT statement in the format of “column:hostvarind”

• -501 THE CURSOR IDENTIFIED IN A FETCH OR CLOSE STATEMENT 

IS NOT OPEN.

Suggestion: Correct logic in application program 

to OPEN the cursor before the FETCH or CLOSE statement.

• -502 THE CURSOR IDENTIFIED IN AN OPEN STATEMENT IS ALREADY OPEN.

Suggestion: Correct logic in application program 

to CLOSE the CURSOR before the OPEN statement.

• -503 A COLUMN CANNOT BE UPDATED BECAUSE IT IS NOT IDENTIFIED IN THE UPDATE CLAUSE OF THE SELECT STATEMENT OF 

THE CURSOR.

Suggestion: Use FOR UPDATE statement in your 

cursor.

• -530 THE INSERT OR UPDATE VALUE OF FOREIGN KEY constraint name IS INVALID.

 Suggestion: Ensure that INSERT row for DB2

PARENT table is completed before INSERT row in CHILD table.

• -532 THE RELATIONSHIP constraint-name RESTRICTS THE DELETION OF ROW WITH RID X’rid-number‘.

Suggestion: Change the 

program to DELETE CHILD table row before DELETE of row on 

PARENT table.

• -551 auth-id DOES NOT HAVE THE PRIVILEGE TO PERFORM 

OPERATION operation ON OBJECT object-name.

Suggestion: 

Contact the support DBA to GRANT the needed privilege.

• -803 an inserted or updated value is invalid because 

the index in index space indexspace-name constrains 

columns of the table so no two rows can contain duplicate values in those columns. rid of existing row is 

xrid.

 Suggestion: Verify DB2 INDEX and, if needed, change the 

statement to an UPDATE.

.NET Framework Versions and Dependencies

.NET Framework 4

Other Versions

• Visual Studio 2008

Each version of the .NET Framework contains the common language runtime (CLR) as its core component, and includes additional components such as the base class libraries and other managed libraries. This topic describes the key components of the .NET Framework versions, provides information about the underlying CLR versions and associated development environments, and identifies the versions that are installed by Windows.
The following illustration summarizes the version history and which versions are installed by Windows.

How to download an .swf file?

This is for Mozilla Firefox:-

1.Find the ADD-ON from tools named-CacheViewer.
2.Now open the webpage having as .swf file and search for that file in the cache viewer.
3.Download the file from there at specified folder.

Hope you will find if helpful.
Thanks.

With regards,
Er. Prachi Bhatnagar.

How to upload Image in Database and Display Image!!

upload images.. first i created a database table with the following columns...

Id --> int (identity column, and primary key)

firstname -->varchar(50)

lastname -->varchar(50)

image -->varchar(50)

and under my web site root directory i have created a folder with name..(images)

here is the code.. to upload the image....



using System.IO;
using System.Data.SqlClient;
using System.Web.Configuration;

protected void Button1_Click(object sender, EventArgs e)
{
if (FileUpload1.HasFile)
{
string fileExt = Path.GetExtension(FileUpload1.FileName).ToLower();
string fileName = Path.GetFileName(FileUpload1.FileName);
string dbfilePath = @"~/images/" + fileName;
if (fileName != string.Empty)
{
try
{
if (fileExt == ".jpg" || fileExt == ".gif")
{
FileUpload1.SaveAs(Server.MapPath(@"~/images/") + fileName);
}
else
{
Response.Write("You can upload only JPG or GIF files...");
}
}
catch (Exception ex)
{
throw ex;
}
}
SqlConnection con = new SqlConnection(WebConfigurationManager.ConnectionStrings["myConnection"].ConnectionString);
SqlCommand cmd = new SqlCommand("insert into images values(@firstname,@lastname,@image)",con);
cmd.Parameters.AddWithValue("@firstname",TextBox1.Text);
cmd.Parameters.AddWithValue("@lastname",TextBox2.Text);
cmd.Parameters.AddWithValue("@image",dbfilePath);
con.Open();
int i = cmd.ExecuteNonQuery();
con.Close();
if (i > 0)
{
Response.Write("Uploaded");
}
}

}