I got a question about PowerExchange throughput the other day. When we were first evaluating PowerExchange as a solution, this was a primary concern of ours. We devoted hundreds of hours designing, implementing, and refining a good test of our throughput. In the end, we over-tested. The throughputs that we dealt with in the test were so much more than our production systems handled, even during peak load, that the actual system handles its load with great ease. Of course, sometimes that’s the point of load testing: to determine theoretical and not actual limits.
Our physical environment
We have a source (PeopleSoft) database running on Oracle 9i/Solaris. A target (ODS) database running on Oracle 9i/Solaris. And we have PowerExchange running on the same Windows 2003 box as PowerCenter. In mid-July, we’ll be upgrading both source and target databases to 10g.
Our data environment
Our current change data capture mappings captures 10 source tables and writes to 30 targets. 3 for each source table. 6 of these tables are considered “wide” tables. These are also the tables with the most DML activity in our PeopleSoft system. One has 23 columns totalling 2.2 kb/row. Another has 148 columns totaling 1.9 kb/row. You get the picture. Lots of data per row. We are capturing every byte in our ODS, plus some extra LogMiner columns like cap_tm.
We don’t have a high, sustained throughput in our system. Our high throughput periods are when batch cycles complete. This is usually nights/weekends when source/ODS latency is not as much of a concern.
Our PowerExchange settings
After discussing our environment with Informatica, we’ve come up with a couple tuning parameters for our system. These are tuned for lower latency, not higher throughput.
- All of our sources/targets in the same mapping.
The fewer open sessions in LogMiner, the faster your reads are going to be. This one was non-negotiable since our primary directive (and reason for using PowerExchange instead of using Oracle triggers) was not to impact the source database. - UOW COUNT
This parameter in the source connection is set to 1. When PowerExchange reads one changed row from the source, it sends it directly to PowerCenter. While I don’t have any specific measurements, I seem to remember dramatically higher throughput processing batch commits when we had this set higher. - Real-Time Flush Latency
This is actually set to 10000ms, which I seem to remember was the default. No significance here, other then that when we did load testing, we used the flush latency and UOW count to balance high throughput with tolerable latency. If you have a generous service level agreement, like “Capture source data within five minutes of the change.” You can up your real-time flush latency and your UOW Count and have PowerCenter process mini-batches of rows. This would really improve throughput.
So, how much throughput?
This varies by table, of course, because the wider tables have lower throughput than the narrower ones. We have bursts of throughput of up to 4700 rows per second. But that number doesn’t really tell you much. 4700 rows of a 23 byte table isn’t much data. We’re not sure if that number is actually updates of our 23 byte table. Actually, I should look into that.
The clearer measure of throughput is data volume or bytes/sec. Not rows/sec.
The most frequent batch job we see happens to update our widest source table. We regularly see throughputs of up to 1200 rows per second in the mornings when this job commits. That’s 2.6 MB/s. Not bad. Our “standard” throughput after batch windows is between 500-1000 rows per second. Which is still between 1-2MB/s.
In summary, our biggest concern in using PowerExchange turned out to work very well. We were able to get by tuning very few variables (just the source connection in the Workflow Manager) without having to look into source/target tuning.
I invite your comments via email scolin@pobox.com or here.
Tags: Informatica, Oracle
To follow up on my previous post about the regression of federal tax rates. Malcolm Gladwell had a brief snippet in his blog today about CEO pay. Brief because the numbers he cites speak for themselves. The problem with the rich-poor gap today isn’t that CEOs are overcompensated but that our tax laws are woefully regressive. High taxes hurt growth? It seems to me that those years in the late 40s-50s were pretty good.
Tags: Tax Law
I scan a handful of financial blogs pretty regularly. I also scan a handful of technical blogs relating to SQL Server and Oracle. It isn’t very often that a blog from one of these worlds links to a blog in the other. But here’s JLP from AllFinancialMatters linking to Don Burleson of oracle-dba.com fame. I guess we’re all just geeks.
Tags: Other
I was listening to a short piece on NPR the other night that discussed tax laws. The claim was that “If conservatives had their way, they would only tax consumption. If liberals had their way, they would only tax savings.”
It didn’t quite seem right, but it left an impression. My understanding of tax law history is that taxes in the early 20th century were very progressive. Before WWII, income was only taxed for the very wealthy. Middle and lower classes paid NO income tax. War-time came and people were willing to pony up to support the effort, and the tax was never repealed.
Capital gains taxes are a similar story. Before the 1970s, capital gains were taxed at some ridiculous rate, somewhere between 70-90%. Since then, they’ve steadily decreased to their current rate today, where long term gains are taxed at 10 or 15% and short-term gains are taxed at your marginal tax rate.
So to hear that conservatives want sales taxes instead of capital gains taxes, it sounded about right to me. It ignores the question of why income taxes are getting more regressive, but that’s another issue.
After reading this letter by John C. Bogle, the founder and CEO of Vanguard, I realize why this idea is so unsettling to me. Capital gains taxes are not a tax on SAVINGS. They are a tax on PROFIT on somebody’s excess money. You can invest in companies by buying stock and that investment is 100% tax-free. It’s when you sell that investment (to spend or re-invest) that it’s taxed.
It is not the tax code that needs changing; it is the “short-termism†exhibited by the vast majority of mutual fund managers, and the short-sightedness of mutual fund shareholders who refuse, out of naiveté or even ignorance, to look after their own economic interests.
I know that he’s not talking about capital gains in general but a more obscure provision regarding the taxation of transactions in a mutual fund holding, but no doubt the same principles apply.
Thanks to JLP at AllFinancialMatters for the link to Bogle’s blog.
Tags: Tax Law
For the past two years, I’ve been listening to John Prine
I haven’t bought many CDs in the last couple years, but I bought his Anthology and have been listening to it almost constantly ever since. Two years and I still keep picking new things out of those same old songs.
Whenever I read about organ donation, I get a chuckle thinking of his “Please Don’t Bury Me” which features clever hooks like “the deaf can take both of my ears if they don’t mind the sight” and “give my stomach to Milwaulkee if they run out of beer”.
Anyway, a couple months ago, I learned that he was playing the State Theater here in Minneapolis. Knowing that I would have a two-month old at that time, I balked at the timing and — with diapers putting a new strain on our budget — the cost. Then I remembered my regrets about never seeing Frank Sinatra or Johnny Cash, and I changed my mind. But by then, they had sold out. Nuts.
We just got a call today from Cities 97, our local Clear Channel affiliate that plays a decent, if predictable, acoustic song list on Sunday morning & evening. We won the tickets to the concert. It’s this Friday.
I am so excited. It’s a small-ish venue and, considering his clever lyrics, I’ll bet he’ll have some good stage patter. And I’m excited to hear more of his music since I only have two disks of his expansive catalog. Not sure where the seats are, but who cares?
Plus this is going to be a great date for me and my tater away from our lil’ tater tot. Two concerts in six months? I almost feel hip again!
Score another run for Fred Wilson’s VC Cliché of the week. He discusses “betting the company” by taking risks instead of managing risk.
I think this is why many (most?) big company managers fail as venture stage CEOs. They are taught to mitigate risks, to plan, to protect. They don’t want to fail and so they do.
Tags: Other
Talk about making your technology easier to use. For a while, I’ve been looking for a solution to our calendaring problem. You see, Julie and I each make commitments for ourselves or for both of us. We commit. Make a mental note. And then either don’t write it down on our calendar. Don’t mention anything to the other. Mention something that we or the other forgets about before the event happens. etc. etc.
We wanted a solution to all this. I was hesitant for it to be a technical solution, because I wanted it to be cheap and really easy to use. I’ll leave alternatives we considered to another post, but we are currently trying google calendar. We both check our gmail frequently, so it’s a pretty natural fit. It is free, but I was still hesitant about it being easy to use.
My pie-in-the-sky dream would be that I could have a conversation over email about getting together with someone and a link would just show up that I could click and automatically add that to my calendar. A lot of calendars will accept iCal files, but to just detect an event from a conversation would be totally cool.
Gmail makes this a reality. Note the link the right of this conversation.
