More Malware Delights

Another Monday, another malware. Today, I am presented with a Windows PC that pops up various virus warnings. An application called Personal AV claims a broad array of infections and all you have to do is buy Personal AV and it will solve them all for you. Nothing we haven’t seen a dozen times before. But, this system was supposedly protected by a commercial antivirus program, to be shamed later, that failed to detect this variant of a tired theme.

I found the culprit in C:\Program Files\PersonalAV\ conveniently named pav.exe. Scanning this file produced no alarms. Indeed, scanning it at VirSCAN.org showed that only 2 of the 38 antivirus scanners caught this file as malware. The two successful ones were Authentium and F-Prot. Kudos to both of these vendors.

Sounds like a new variant, doesn’t it? Well, that might have been the case except that Google searches produce at least one month old results for this particular malware. It’s not new yet none of the major antivirus vendors detect it! A sad testament for the anitvirus industry but, probably not all that surprising to anyone in the IT industry.

So why then shame a particular one when they are all guilty? I’ll tell you. I’m shaming CA eTrust Integrated Threat Management 8.1 because I went to the trouble of trying to submit the virus via their web based virus submission page and they violated my privacy, wasted my time and failed to take the submission..

It was bad enough that CA’s virus submission page required personal details such as first and last name, telephone number and email address in order to submit a virus sample. But then after surrendering my personal information to their marketing database and wasting my time and knowledge in providing them with the sample virus and information on which files and registry keys are involved in removing it, I am presented with a page that reads:

An error has occurred while submitting your virus sample. Please try again later.

In order for me to do their job, I have to surrender personal information, waste my time AND do it again? Bad job Computer Associates!

For those that encounter this particular bugaboo, here’s how to clean it up:

Reboot Windows into Safe Mode.

Delete C:\Program Files\PersonalAV and the pav.exe within.

Delete %systemroot%\System32\msxmlm.dll if it is between 365KB and 370KB in size. There is a legitimate Microsoft dll of that name that is 404KB in size.

In the registry delete the following entries:

HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Explorer\Browser Helper Objects\{A77D3539-581D-450C-9E44-A84C415A6172}

HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Run\PersonalAV

HKEY_CLASSES_ROOT\CLSID\{A77D3539-581D-450C-9E44-A84C415A6172}

Restart your computer and update your antivirus signatures. Hopefully they will be detecting this malware by the time you are reading this.

What Just Happened?

What just happened? Again!

There I was happily surfing along, minding my own business. Suddenly, Xorg, the windowing system on my Linux desktop just crashed, taking all my applications with it. No sooner had I uttered “What the [Explitive Deleted]” had Xorg and KDM restarted. I was quite startled by this event because I was so unaccustomed to anything like it. This is only the second time that this has ever happened on this years old system. It is also the second time that this has happened in the last week. Twice in a week!

I was doing the same thing on both occasions so, I naturally have a suspect. The suspect is Firefox 3.5.x. Both times that Xorg has crashed and restarted I was surfing the net with Firefox with a few (less than five) tabs open. The first time it happened I was running Firefox 3.5.0. This time, I was running Firefox 3.5.1.

Naturally, I checked the various logs to try and see what the cause was but, the only mention in any of the logs is

kdm[3415]: X server for display :0 terminated unexpectedly

in /var/log/messages.

Contemplating these two incidents, it does remind me that I had occasionally noticed and ignored images from inactive tabs showing through onto the active tab like a video artifact. It was a minor issue and I easily ignored it but now, I’m sure they are related.

Here’s an example of the video corruption.

No, I’m not running Compiz or any sort of 3D accelerated desktop tripe. This is a basic business class desktop that has been working fine for years and has had zero software or configuration changes except for a new copy of Firefox 3.5.x and a new copy of AdBlock+ 1.1 to go with it.

It looks like Firefox 3.5 has some more issues than those that were fixed by 3.5.1. The problem hasn’t yet been severe enough for me to dump Firefox 3.5.1 but, this is troubling. Stability is becoming a critical feature in the browser war and having a browser crash Xorg is a serious issue. Lets hope that this doesn’t continue and that Mozilla get it fixed before they get a bad name for themselves.

Update: It’s happened again! In fact, I have found that I can force the crash to occur pretty reliably. Take a JavaScript heavy page, scroll to the bottom, grab the scroll bar and rapidly scroll back to the top. Kaboom!

A newly discovered bug, reported by SANS, makes me wonder of they are related.

HP Printers Race Downhill

For well over a decade, Hewlett Packard has been THE premium printer manufacturer. Network administrators the world over have sworn by the reliability and durability of the HP LaserJet line. Those somewhat pricey behemoths cranked out page after page for years on end with nary a problem beyond regular maintenance. But, HP seems to be incapable of maintaining the status quo. They have felt the need to tinker and “improve” their printers for the past few of years. The result is a sad state of affairs, to say the least.

Today’s HP printers are a whole new breed. In fact, it almost seems like HP introduces a whole new breed of model numbers every six months or so. Each iteration seems to be a little cheaper and a whole lot less durable. Which is a truly sad condemnation for what was long regarded as the industry stalwart.

But, much worse than the printers themselves is the supporting software. I hesitate to call them drivers anymore because the supporting software has become so very much more, whether we wanted it or not.

It use to be that HP offered a small driver to interpret an application’s output and format it into the printers native language, in this case Printer Control Language(PCL). Though there were always exceptions and some compatibility issues, these drivers mostly just worked. A user clicked print, the printer whirred, the printed page came out and business moved along.

That was then, this is now. Today we have “drivers” whose download size is pushing past the 15MB mark and in some cases over 300MB! As well as occupying a lot of space they also feel the need to make themselves known on a regular basis with frequent pop-ups and colorful text balloons, whose window frame breaks from the Microsoft Windows standard, every time a page is printed.

And, lest you run out of ink/toner, these drivers will start nagging you to buy more when you get down to 30% remaining. Or is it 50%? But wait! There’s more! It’s not enough to alert you that the toner cartridge may need replacing in a month’s time, HP also thinks that you would really enjoy shopping for their other products like paper, cables, cameras, and what-have-you. All the while, the business of actually printing a page takes a back seat.

HP heard their customers anguish about the growing number of different printer drivers, not to mention HP’s own anguish in developing all of them, and struck on the genius idea of a universal printer driver. One driver to rule them all! The printers I mean. It made so much sense that no one believed it was possible. Well, perhaps they were right to think it impossible. There are now dozens of different “universal” drivers for dozens of different models of printer and, to be honest, I have yet to get a universal driver to work! It must be me but, after all these years, I thought I knew how to set up a printer. I even read and followed the instructions!!! Can you imagine? Still no luck getting a model specific “universal” driver to even print a test page on Windows XP!

Most unfortunately for the users of today’s HP printers, all the extra software stuff slows down their computers. It causes all sorts of incompatibility issues, lockups, difficulty or inability of administrators to push out point and print drivers. The list goes on and on. Congratulations HP, you’ve done a terrible job.

Update: I managed to get the Universal PCL5 driver to work with a network connected P4014n. That’s what produced the print job balloon shown above. But, this driver would only work when connected via a standard IP port monitor. It would not work for a USB connection to the printer.

Update: Update: I spoke too soon. The Universal PCL5 driver appeared to work for this printer. But, it fails to work when printing WordPerfect documents that were created with a different default printer. Although, it does print new WordPerfect documents, created with the Universal PCL5 driver as the default printer, properly.

The final solution was to configure the printer with a LaserJet 4200 Post Script driver. It’s the only one that seems to print everything properly, regardless of application or document origin.

Firefox 3.5 Performance

Firefox 3.5 has been out for a couple of weeks now and, for the most part, it has been treated like a rock star. All the buzz is about how much faster Firefox 3.5 is than anything else, or at least how much faster it is than Firefox 3.0.

I’ve always complained about the performance and resource consumption of Firefox. While it is still my preferred browser, it has always seemed slower than IE, Konqueror and Safari. So, when everyone proclaimed how very much faster Firefox 3.5 was, I had to give it a try.

I can’t easily have multiple versions installed on a Windows machine but, I can very easily have as many different versions as I like on Linux. So, I grabbed a copy of Firefox 3.5 and put it on a Linux installation that I had that also has Firefox 2.0 installed. I’m now able to easily test either one of them and compare their performance from site to site.

What I’ve found has shocked the Net Codger. Firefox 3.5 is not noticeably faster than Firefox 2.0 on the same Linux platform! While articles about 251% increases in speed are being bandied about, I see little if any improvement in performance over a much older version. What’s all the noise about? “Where’s the beef?”

Even Mozilla’s own test results (above) claims much higher performance over previous versions of Firefox. At least on Windows. So, why the disparity with my testing.

What I have found is that while it seems that rendering of HTML pages isn’t significantly improved, when you hit JavaScript heavy pages, things start to look quite a bit different.

I performed the SunSpider JavaScript test used in the above graphic on my Linux machine and there was indeed a multifold increase in java script performance. There was also a complete absence of pop-up warnings regarding the scripts hanging the PC, as with Firefox 2.0. Even the WordPress admin console, used to compose this post, was much smoother and faster when using Firefox 3.5.

There are many other improvements in Firefox 3.5 over previous versions as well. Not the least of which is a reduction in memory consumption that I greatly appreciate.

After initially being disappointed with an apparent lack of improved performance, I am now quite impressed with the much improved performance of Firefox 3.5. Now I have to try it on Windows.

Smart Grid – Miracle Cure Or Snake Oil?

The hyperbole for Smart Grid technology is becoming quite a fervor. The greenies espouse that that it will save the planet and the corporations pitch that the technology will change the very being of technology itself. Smart Grid, it seems from all the hype, will part the waters, cure cancer and possibly even save the automobile industry. But, none of the media craze seems to say exactly what Smart Grid is or how it will do all the miraculous things that they predict.

The U.S. Department of Energy(DOE) seems to have hatched the Smart Grid idea, or at least the moniker. They describe what it is in their non-technical guide. The guide is a 48 page “laymans” definition of what Smart Grid supposedly is.

The 48 page guide is actually a marketing brochure that even Cisco or Microsoft would be envious of. Its buzzword compliance is truly impressive. But, not until page 18 do we finally get down to what Smart Grid really is. At which point it finally comes clean and says:

It gives grid operators far greater visibility into the system at a finer “granularity,” enabling them to control loads in a way that minimizes the need for traditional peak capacity. In addition to driving down costs, it may even eliminate the need to use existing peaker plants or build new ones – to save everyone
money and give our planet a breather.

That’s a lot of cost saving opportunity for the utilities. And not having to build more of those pesky power plants really reduces their costs and workloads too. But, just in case someone wants to beleaguer the “give the planet a breather” excuse at the end, we need to look at page 19 where it tells us what smart grid is NOT.

Further clarification: Devices such as wind turbines, plug-in hybrid electric vehicles and solar arrays are not part of the Smart Grid. Rather, the Smart Grid encompasses the technology that enables us to integrate, interface with and intelligently control these innovations and others.

In other words, it is not new generation, new generation technology, clean or green generation, increased generation or anything to do with generation. It is all about controlling the distribution and, less discussed, the consumption of the electricity. And controlling the consumption is the little discussed portion that greenies and unwary consumers will finally see too late. It’s all about the utilities’ ability to turn off your appliances when they see fit. But, the marketing spin presents it all in such a friendly manor. Who wouldn’t want:

real-time pricing – These are energy prices that are set for a specific time period on an advance or forward basis and which may change according to price changes in the market. Prices paid for energy consumed during these periods are typically established and known to consumers a day ahead (“day-ahead pricing”) or an hour ahead (“hour-ahead pricing”) in advance of such consumption, allowing them to vary their demand and usage in response to such prices and manage their energy costs by shifting usage to a lower cost period, or reducing consumption overall.

Page 22

I’m sure that you are like me and are eager to have the option of an hour’s notice to turn off your air conditioning in the middle of a hot summer’s day or pay a newly imposed higher and previously unknown rate. What? you don’t see that a a benefit to you so much as to the utilities? Do you hate the planet or something? Remember, this is the Smart Grid. It’s better for us and the planet! They keep telling us so.

On page 23 it tells us that “there’s a lot that consumers can do to help the grid.” While I was foolish enough to believe, for years, that the grid was supposed to help consumers. I foolishly believed that the electric utilities were supposed to benefit the consumer and now I find out that it is the consumer that is supposed to benefit the utilities. And now, thanks to Smart Grid, we can!

Smart Grid, if you hadn’t noticed, is a marketing campaign designed to generate massive funding for the companies in the industry. Its purpose is to vacuum out grants and multi billion dollar government economic stimulus packages, as well as private investment funds. There is little or no environmental benefit. There is little or no increased electrical generation or capacity.

Open Office Writer & Regular Expression Searching

Open Office Writer And Regular Expression Searching

It’s always the simplest things that take the most of my time. About once each year I find a need to remove lines or, more accurately, paragraphs from Open Office Writer documents. It’s simple enough really, Each line separated by a carriage return is a paragraph. I want to find lines or paragraphs that begin with a particular word or code and replace the entire thing with something else or perhaps even delete it completely.

I’m quite sure that there are lots of fancy ways to do this using macros but, in my mind it should be a simple search and replace. Despite the simplicity, I struggle each time because the length of time since last I used the technique causes me to forget how to do it.

But it really is quite simple. Start by bringing up the Find & Replace dialog box and click the More Options button.

After clicking the Regular expressions tick box, it’s a very simple regular expression formula or recipe to get the job done. In this case we are searching for a particular word or code at the beginning of the line or paragraph so, we begin with a ^ (caret) to designate the beginning of the paragraph. This is followed by the string that we are searching for, UnwantedText in this case. Then we append a .*$ at the end. The .* (dot asterisk) matches any string of characters and finally the $ (dollar sign) matches the ending paragraph symbol.

^UnwantedText.*$

That matches the entire line or paragraph and we can then replace it with whatever we wish. Or if removal is your goal, replace it with a blank (null, nothing) in the Replace with field. Poof all gone.

VMWare ESX3i – Missing Datastore

You may have experienced it yourself but, back in August of 2008, VMWare had a rather major incident with a patch/update that rendered Virtual Machines(VM) the world over completely incapable of powering on. It turned out that due to a bug in the licensing portion of the update’s code VMs that were running would continue to run and VMs could be rebooted as well. But, if a VM was powered down, later attempts to power it back up resulted in a licensing error and a VM that stayed powered off.

This unexpected event scared the daylights out of me and thousands of other VMWare administrators around the world. For a few very tense hours, no one knew what was happening. Then we figured out that if you changed the date on the host server to the previous day’s date, you could finally power on your VMs and the panicked support calls started to subside. A few days later, VMWare released a new update that properly fixed the issue and life went on.

But, since that dark day, I’ve had a couple of more incidents when patching or rebooting VMWare hosts and I’ve lost my blind faith in VMWare’s quality control. This has caused me to delay and or stop patching VMWare hosts, unless absolutely necessary, and I now always test prior to applying patches and updates. Sure, I should have been testing all along just like all my other vendors releases but, VMWare was so good up until that dark day that the Net Codger and so many others could not imagine a VMWare product causing a problem.

It’s update time.

I needed to bring a few VMWare servers up-to-date so, it was time to do some testing. It so happens that I have a little VMWare ESX3i test server sitting under my desk. It’s just for testing so it is running on a Shuttle SG33G5. This tiny box has a Quad Core processor, 4Gb RAM and a 1TB SATA hard drive in it. It is not supported by VMWare but, I made it work pretty easily and I usually have 10 or 15 VMs on it with several in use at any one time. It’s small, quiet(one of the most iportant aspects), fast enough for testing and light use by 5 people or less at a time. It’s also perfect for testing VMWare patches and updates prior to production deployment.

So, I fired up my VMWare Updater application, downloaded a few hundred megabytes worth of updates and installed them. All went well and I restarted the VMWare host. The host server started back up just fine but, there were no VMs. I mean none. The data store was gone! The inventory list of VMs showed

Unknown 1 (inaccessible)
Unknown 2 (inaccessible)

and so on. Foiled again! But, I was glad to know that this was a relatively unimportant test machine and not a production server with users wondering why it was down. Still I couldn’t help but resent the idea of having to recover/restore/rebuild my test VMs. I just don’t have time for this stuff.

Poking around in the VMWare Infrastructure Client I saw that the storage adapters were present so, it didn’t seem to be the SATA storage driver issue that I had first suspected. The storage adapter was there and the disk was there but no datastore. Rescanning the storage adapters didn’t make it any better either. Under the storage configuration, I started to step through the add storage option and that listed all of the partitions on the disk including the VMFS partition that housed the missing datastore but, it also threatened to delete all the partitions if I proceeded so, I knew that wasn’t the right direction.

Making my way over to the Events tab, I found this entry:
May 2 11:41:56 boz101 vmkernel: 143:30:02:49.611 cpu4:1040)ALERT: LVM: 4482: vml.4a043bd5-a7e343be-e3b6vmhba0:0:0 may be snapshot: disabling access. See resignaturing section in SAN config guide.

Reading up on this message lead me to the Advanced Settings LVM page. Here I enabled LVM resignaturing by setting the Value in LVM.EnableResignature from 0 to 1. Then, A rescan of the storage adapters found a storage volume called SNAP-26DBAABD-VMHBA0:0:3. I still wasn’t quite there but, at least I had a volume showing up again.

I renamed the volume to the original “Datastore 1” but, the VMs were still showing up as unknown so, I set the LVM.EnableResignature setting from before back to “0” and I restarted the host in the hopes that it would return to operation.

When the VMWare server came back up, my data store was there and its contents were intact but, the VM inventory list still showed
Unknown 1 (inaccessible)
Unknown 2 (inaccessible)
etc. and none of my VMs had started.

To resolve this issue, it was necessary for me to remove each Unknown 1 (inaccessible) from the inventory list and then, using the datastore browser, open each VM directory, right click the *.vmx file and choose add to inventory. It’s a tedious process but, its simple and fairly quick.

Finally, with my VMs added back to the inventory, I could power them up. But, wait, there’s still one more step.

When each VM is first powered on, it pauses and there is an error message stating that

5/2/09 12:18:45 PM, Message on Windows XP Pro on esx.local in ha-datacenter: The location of this virtual machine's configuration file has changed since it was last powered on.

If the virtual machine has been copied, you should create a new unique identifier (UUID). If it has been moved, you should keep its old identifier.

If you are not sure, create a new identifier.

What do you want to do?

The proper step here is to right click on the VM in the inventory list and choose “Answer Question” and then choose “Keep”. The VM finally starts as normal and you’re back in production. You must also go to the Virtual Machine Startup/Shutdown configuration screen and re-establish your automatic startup delays and startup order for your VMS if you want them to start automatically when the host is started.

The fact that this test machine is not the same as those in production and that it is not supported hardware does not escape me. I realize that I was begging for trouble. But, now I know that I should expect trouble and I know how to fix it. Going through similar motions on a production system with 40ish VMs on it will likely take me an hour so, I will plan for downtime and fun. If I don’t have problems, then all the better.

Ubuntu 9.04 – What’s All The Hubbub, Bub?

Ubuntu 9.04 was released this week and the Net Codger thought he’d give it a try. There had been a few articles prior to its release that seemed to suggest that there were big user interface changes in this one. Most often cited were a new notifier and a redesigned shutdown button/menu.

I had a codgerly Dell Optiplex 745 system that was already running Ubuntu 8.04, the Long Term Support(LTS) release. It hadn’t been updated to 8.10 because of the perceived benefit of LTS, stability. But, the stability wasn’t real. Over the coarse of time, security updates had degraded the system and it had developed some really annoying problems that I just couldn’t seem to fix. Not the least of which was th inability to shutdown the system by clicking the shutdown button. When I tried to shutdown this way, the Gnome desktop would start to close and then the system would freeze with a blank desktop. Nothing I tried would allow a software shutdown on this system that had worked properly after the initial 8.04 installation. Pressing the power button immediately initiated a proper and complete system shutdown.

After deciding to upgrade to 9.04 hoping to solve my problems and see what the new version was all about, I fired up Update Manager. Ubuntu doesn’t allow you to upgrade and skip versions so, I first had to upgrade my 8.04 to 8.10. I clicked the upgrade button and went to bed. The next morning, I had 8.10 up and running. That’s one great thing about Ubuntu and other Debian based systems, package management and upgrades are dead simple. I didn’t bother testing anything in 8.10, I was interested in 9.04 and immediately started the upgrade process.

Midway through installing the new 9.04 packages the upgrade process crashed. Attempts to rerun the upgrade failed with all sorts of errors and I decided to restart the system. Huge mistake! Upon reboot, Xorg would not start and attempts to use low resolution mode and to reconfigure Xorg all failed. It was clear that some but not all of the files had been replaced and I now had all sorts of version conflicts.

Switching to a console screen, I was able to get networking started and then used apt-get with the force switch to restart the upgrade process. This time, the upgrade completed successfully. I was very satisfied to see the newly upgraded system restart and come up properly without any errors.

I was immediately greeted by a newly designed login screen. It was attractive but really, it’s only a re-skinning so it’s not that impressive. Likewise, there was a new default desktop wallpaper which I thought was attractive but, Mrs. Codger immediately changed it to the elephant skin wallpaper of Ubuntu 7.10. I found that interesting as Mrs. Codger had not changed the default Hardy Heron wallpaper in a year of using 8.04.

After logging in, I experienced my first “notification”. Now in some of the articles that I had read about the new notification system, it was compared with Apple’s system. But, my first thought, upon seeing it, was that it was a crude looking version of Microsoft Windows balloon notifications. Ugh! That’s certainly not a feature that I had hoped for.

Using the system over the next couple of days left me wondering just what had changed. The new 9.04 system seemed almost no different than the 8.04 that was running a couple of days before. There had been talk of massive performance improvements but, I didn’t notice any. The “big changes” that I had anticipated were nowhere to be found. Everything pretty much looked and felt as it did before. The only thing I really appreciated about it was that my newly redesigned shutdown button was working again.

On the down side, promised improvements to video and Compiz were a big disappointment. This machine uses the very common Intel GMA 3000 Q965 integrated graphics chip that has been around since 2006. There are community Xorg drivers for this chip as well as Intel’s own, now open source, drivers for it. But, more than two years on, Ubuntu still has the chip “blacklisted” for Compiz.

Forcing the issue and turning on Compiz or any of the desktop effects still cause video artifacts and overlay problems that are quite annoying. And, as before, Google Earth is completely unusable due to video problems when any desktop effects are enabled. The only way to use Google Earth is to set desktop effects to none.

That’s really annoying! The much ballyhooed, and now standard in Ubuntu, Compiz remains a disaster on this video chip and I can no longer accept any excuse for this deficiency.

Another issue was the file indexing service. After running for a couple of hours, it stopped and popped up an error about the index being corrupt. It asked if I wanted to re-index everything. I tried this and it again failed after a couple of hours. I then tried deleting the indexes and restarting the service in the hope that a newly created index would be the answer. Again, it failed after a couple of hours and I finally disabled the indexing service altogether. This annoyance wasn’t too disappointing though, I have a dislike for all of the presently available desktop search/indexing services available for all of the Oses. I find them obtrusive, slow, ineffective and a major drain on system performance.

Overall, Ubuntu 9.04 seems like a solid distribution. It has all the great benefits that Ubuntu has offered including ease of use, attractiveness, simplicity and even some stability. But, I just don’t see that much difference between 8.04 and 9.04. I’m glad to have my power button working again but, for me there was no other meaningful improvement. Certainly, it should not have been necessary to upgrade the entire distro in order to repair a feature broken by a security update in an LTS version. Additionally, support for a very widely deployed three year old chipset, for which the manufacturer has released open source drivers, doesn’t seem too much to ask. But, Ubuntu again failed to deliver.

On one hand, I want to beat up Ubuntu for coming so close and failing to deliver a truly fantastic desktop OS. On the other hand, based on recent experiences with openSUSE and a few other distributions, Ubuntu is definitely the desktop Linux distro with the least fuss!

I suppose the really big question is whether or not Ubuntu 9.04 is better than the impending Windows 7 release. That’s a tough one to answer as they each have a lot of unique pros and cons. But, right now, I feel that Windows 7 is ahead of Ubuntu and the fact that Windows 7 hasn’t even been released yet says a lot! Ubuntu really needs to stop futzing with shutdown menus and start doing some real work. Refreshing the desktop and improving hardware support are both critical needs for 9.10 and one or two “killer apps” are desperately needed to keep Ubuntu as a meaningful contender.

Online Console Games Should Have Expiration Dates

More and more games today offer online game play. There’s no doubt that playing with a group of live players makes the game a lot more fun than the old single player against the machine model. Live players are much less predictable than the machine and varying skill sets make for an interesting mix of fun and challenge. The ability to interact with the other players in game or via text or voice is an additionally interesting and engaging aspect. But, in the case of the console games, these online scenarios are almost always tied to the vendor’s own game servers. In most cases, there is no way to use any other game server.

This vendor server lock in creates an issue when the vendor loses interest in supporting the game or goes out of business. In this case, the game essentially dies because if you don’t have the servers to connect to, you can’t play online and more and more games today are of the online only type. This means that when the servers go away so does the game. It doesn’t matter that you want to play the game you paid for and love, it’s gone!

This game death works well for the vendors. It essentially forces the game players to upgrade to the newer versions or buy other games whenever the vendor no longer feels like supporting the game or when they decide they want more money from the gaming community.

One example of this is unfolding as I type with Electronic Arts(EA) and the immensely popular Team Fortress 2 game. Team Fortress 2 was licensed by EA for the PlayStation 3(PS3) console and was release as part of the Orange Box set in November of 2007. Despite being hugely popular the revenue stream from the game for EA has whithered resulting in an abysmal lack of support. EA’s servers have been severely lacking in maintenance for a long time and their North American servers have been completely missing in action for months. With only European servers still alive, North American players suffer awful game play due to latency (lag) issues and the game is virtually unplayable. And, since there is no alternative to EA’s game servers, the PS3 version of Team Fortress 2 is all but history for North American players. When the European servers die, no one will be able to play the game at all anymore since it is online only.

EA doesn’t care because they got their money for the game up front. Running these servers after the initial surge in game sales is just an expense that they want to cut. They also look forward to another surge in sales of some other game when they shutdown the servers and render the game unplayable. Too bad if you paid for it or like playing it. It’s over and the game is gone!

The Net Codger doesn’t like that. Having spent the money for a game I want to be able to play it whenever I desire. I can still break out the Nintendo 64 that I bought back in the 80’s and play Super Mario Brothers or Duck Hunt if I want to play them. But, in the case of the PS3 Team Fortress 2, it’s seems to be all but over after only 17 months! In ten years, no one will be able to play Team Fortress 2 on the PS3 at all.

Now, Team Fortress 2 is just an example. There are many games out there that are just the same and face the very same fate. Some have already gone this route and will never be played by anyone again. Any game that relies on vendor servers with no way to use alternative servers, is at risk of not just obsolescence but vaporization.

While some people, like EA, might argue that that is the way it is or has to be, that is obviously not the case. The ability for players to setup and run their own servers, as is the case with many PC based games, allows the game to live on indefinitely regardless of the vendor. But, the vendor’s don’t want you playing old games. They want you buying new games and, with the ability to turn off the old games, they do not have to make the new ones compelling. They don’t have to be innovative with the new games when the can simply pull the plug on the old ones and force you to buy a new game.

Some people say that this is why games and software in general should be sold on a subscription basis. Indeed this seems to be the way the vendors are positioning themselves for the future. But, the Net Codger doesn’t like the subscription model at all and it seems that most other people don’t either. Despite the desires of the software industry, people still want to pay a fixed one-time cost for a product as property and then exercise ownership of that property going forward. It’s not unreasonable to feel that way since, that’s how most of the world works. Sony doesn’t charge rent for the PS3(yet) and people don’t want to pay rent for their games.

So far no one seems to be talking about this industry trend. But, I can’t help feel that the resentment will build and that I am not alone. At the very least, I think that the vendors should be placing an expiration date on the box of their software. That way people will know when the servers will be retired and the game will be rendered unplayable. But, that would shorten the sales cycle of the game and the vendors aren’t likely to do it.

BGP Configuration for AT&T MPLS Service

A friend that I use to work with called me the other day. He was anxious about an upcoming network installation where he would be utilizing Multi Protocol Label Switching(MPLS) to connect several branch es of his client’s offices. Welcome to the 21 century.

His anxiety was due to the fact that this was his first MPLS encounter and he didn’t have a clue what was involved. Furthermore, the MPLS service provider, AT&T, hadn’t yet provided any details about the MPLS connections. He asked me if I could bring him up to speed on MPLS.

He wasn’t asking for much, was he? He simply wanted me to condense years of training and experience on a rather convoluted and complex subject into a 5 minute phone call. The Codger gets these types of calls all the time and, believe it or not, MPLS is just a little bit more complicated than what can be explained in a 5 minute phone call. But, like I said, this was a friend so, rather than explaining how he can pound sand, I agreed to email him some details instead.

As promised, I emailed him several pages worth of brief descriptions of how MPLS worked and different possible configuration scenarios. Combined with some links to technical documentation on Cisco’s website, I felt that he was well on his way to building a MPLS network of his very own.

When he called back a few days later, after reviewing what I had sent him, I could just about hear the confused blinking stare over the phone. I explained again that it was pretty unusual for a retail service like AT&T’s MPLS offering to actually require any MPLS configuration on the customer’s end. He couldn’t get his head around the fact that the random and excessive use of the MPLS buzzword is just marketing, it’s not what the end user has to work with. Usually, MPLS service providers just give you a circuit termination point, ethernet, frame-relay, serial PPP. You dump your data into it at one end and it comes out the other. No MPLS worries for you.

When my friend gave me the (incomplete) configuration information that AT&T had provided, it was immediately clear that he needed no help with MPLS. What he actually needed was a very basic Border Gateway Protocol(BGP) configuration to connect his branch offices over AT&T’s MPLS network.

I started searching for a couple of good links to configuration examples for him but, I didn’t find any really good ones that fit his scenario and explained what was happening. Most of the tutorials and examples were for configuring BGP to connect your router to two different ISPs on the internet. This was a little different. Simple explanations of BGP are a bit lacking and the fact is that with multihomed internet connections and ever increasing use of MPLS backed Wide Area Networks(WAN), use of BGP is rapidly increasing. Where people used to use frame-relay circuits and static routes or basic routing protocols like RIP, today they are using circuits with MPLS back haul and BGP routing.

With all those BGP newbies now using “scary” BGP routing for the first time. I decided to write up a configuration example that would explain how to simply connect a few branches over AT&T’s MPLS service and also how to setup a dual ISP internet connection. That way we can see both methods for the sake of comparison.

I’m breaking up the configuration into two parts. In this first part, I’ll cover the private WAN configuration. In the second section, I’ll cover the BGP connections to the two ISPs.

Looking at the diagram below, you can see that we have three interconnected branch offices one each in New York, Chicago and Dallas. They communicate with each other over AT&T’s MPLS service and utilize BGP to handle most routing information.

Pink routers are part of AT&T’s MPLS router cloud. The PE designation means Provider Edge(PE), typical of Cisco’s vernacular. Blue routers are those owned by the end user’s company. For the purpose of this article, they are “our” routers and are under our responsibility for configuration and administration.

Dallas

We’ll start with the Dallas branch configuration. Being a stub network connected by a T1 Point-to-Point Protocol(PPP) connection, it is the most simple configuration of the three in this example.


!
! This sets the hostname of the router.
hostname Dallas
!
! Sets the password for privileged level configuration.
enable password [password]
!
! Sets the router's time zone.
clock timezone CST -6
! Turns on automatic daylight savings time adjustment. U.S. DST settings.
clock summer-time CDT recurring 2 Sunday March 2:00 1 Sunday November
! Allows the use of zero subnets when using CIDR. Preserves IP address space.
ip subnet-zero
!
!
! Prevents the router from trying to resolve DNS hostnames entered at the command prompt.
no ip domain-lookup
!
! Enters configuration mode for that interface.
interface FastEthernet0/0
! Duh?
description connected to Dallas LAN
! Clearly, the IP address of this interface.
ip address 10.3.3.1 255.255.255.0
! Enables the interface, making it active.
no shutdown
!
! Enters configuration mode for the first serial interface.
interface Serial0/0
! Duh?
description Connected to MPLS Cloud
! The IP address fro this interface.
ip address 12.87.11.77 255.255.255.252
! Indicates that we will use PPP encapsulation for traffic on this serial interface.
encapsulation ppp
! This prevents redundant directly connected routes in the routing table.
no peer neighbor-route
!
! These commands configure the integrated CSU/DSU on the T1 WAN card.
service-module t1 linecode bz8
service-module t1 framing esf
service-module t1 timeslots 1-24
service-module t1 remote-alarm-enable
service-module t1 fdl ansi ! Ironic for an AT&T service. But still the best choice.
! Turns off Cisco Discovery Protocol on this interface.
no cdp enable
! Enables the interface making it active.
no shutdown
!
! This enables the BGP routing engine and assigns this process
! the autonomous system(AS) number 65001. This AS number
! is private to our network and was arbitrarily chosen.
router bgp 65001
! Disables IGP route validation. Makes things simpler and speeds route propagation.
no synchronization
! Logs changes in the status of the peer router. Link up/down etc.
bgp log-neighbor-changes
! Advertise this network via BGP
network 10.3.3.0 mask 255.255.255.0
! Establishes a peering session with this router and its AS number.
! This AS number belongs to and is provided by the MPLS service provider.
neighbor 12.87.11.76 remote-as 99999
! This prevents classful summarization of routes. Without it, our
! various 10.x.x.0 networks would all appear as a single 10.0.0.0
! route in BGP advertisements.
no auto-summary
!
! Enables CIDR routes.
ip classless
! Sets the default route for this router.
ip route 0.0.0.0 0.0.0.0 12.87.11.76
!
! Creates five telnet terminal lines.
line vty 0 4
! Sets the telnet password.
password [password]
! Prompts telnet sessions for a password.
login
! ! Indicates the NTP time server on the network.
sntp server 10.1.1.87
! Ends the configuration. Duh?
end
!


As you can see, Dallas is rather straight forward. It has an ethernet interface for the LAN and a serial interface, running PPP, for the WAN. There are no MPLS commands to worry about. AT&T is handling all of that for you. BGP is used to propagate our routes from site to site via the MPLS network.

Chicago

Now let’s look at the configuration for the Chicago branch. It is very similar to the Dallas branch but, it’s a bit larger. As you see in the diagram this branch has two LANs, each on different floors of the building. The second floor router at Chicago uses a single default route to connect to the network. We’ll add a couple of commands to the Chicago branch router for this second LAN.


!
hostname Chicago
!
enable password [password]
!
clock timezone CST -6
clock summer-time CDT recurring 2 Sunday March 2:00 1 Sunday November
ip subnet-zero
!
!
no ip domain-lookup
!
interface FastEthernet0/0
description connected to Chicago 1st floor LAN
ip address 10.2.2.1 255.255.255.0
no shutdown
!
interface Serial0/0
description Connected to MPLS Cloud
ip address 12.80.21.13 255.255.255.252
encapsulation ppp
no peer neighbor-route
!
service-module t1 linecode bz8
service-module t1 framing esf
service-module t1 timeslots 1-24
service-module t1 remote-alarm-enable
service-module t1 fdl ansi
no cdp enable
no shutdown
!
router bgp 65001
no synchronization
bgp log-neighbor-changes
network 10.2.2.0 mask 255.255.255.0
! Tell the BPG peers that we also have a 10.4.4.0/24 subnet
! accessible through this router.
network 10.4.4.0 mask 255.255.255.0
neighbor 12.80.21.14 remote-as 88888
no auto-summary
!
ip classless
ip route 0.0.0.0 0.0.0.0 12.80.21.14
! Route to the second floor.
ip route 10.4.4.0 255.255.255.0 10.2.2.2
!
line vty 0 4
password [password]
login
!
sntp server 10.4.4.6
end
!


As you see, we added the command network 10.4.4.0 mask 255.255.255.0 to the BGP section. This causes BGP to advertise this subnet to the rest of the network. If we had multiple such networks, we could substitute the multiple network commands with a single redistribute static command in its place.


router bgp 65001
no synchronization
bgp log-neighbor-changes
network 10.2.2.0 mask 255.255.255.0
neighbor 12.80.21.14 remote-as 88888
! Tell our BGP peers that we also have routes to the networks
! that we specify static route statements to.
redistribute static
no auto-summary


Also note that the neighbor remote-as number is different from the one used in the Dallas configuration. The difference is simply due to Chicago branch entering AT&T’s network from a different point that happens to be in a different autonomous system. Again, this number is provided by the service provider and we don’t have to worry about it beyond this configuration statement that establishes the peering between our router and theirs.

New York

Finally we come to the configuration of New York’s router. It’s a little different than the other sites. It has no serial connections. Instead it utilizes a Metro Ethernet connection to the MPLS cloud. New York is also the site through which everyone connects to the internet so, it has a couple of additional commands in its configuration.


!
hostname NewYork
!
enable password [password]
!
clock timezone EST -5
clock summer-time EDT recurring 2 Sunday March 2:00 1 Sunday November
ip subnet-zero
!
!
no ip domain-lookup
!
interface FastEthernet0/0
description connected to NewYork LAN
ip address 10.1.1.1 255.255.255.0
no shutdown
!
interface FastEthernet0/1
description Connected to MPLS Cloud
ip address 12.91.117.25 255.255.255.252
no peer neighbor-route
no cdp enable
no shutdown
!
router bgp 65001
no synchronization
bgp log-neighbor-changes
! We're anouncing that this router has access to the default route.
! This is actually a default setting that has been entered here for clarity.
network 0.0.0.0
network 10.1.1.0 mask 255.255.255.0
neighbor 12.80.21.14 remote-as 99999
! Here we tell the peer that we have the default route.
! Normally BGP does not propagate this information
! even with the network 0.0.0.0 as above.
neighbor 12.80.21.14 default-originate
no auto-summary
!
ip classless
ip route 0.0.0.0 0.0.0.0 10.1.1.254
!
line vty 0 4
password [password]
login
!
sntp server 10.1.1.87
end
!


In the New York configuration our MPLS connection is FastEthernet rather than serial PPP but, you can see that there is little difference. Regardless of what type of layer two interface we use, we only have to configure IP and a routing protocol. There are no MPLS specific settings that we have to worry about.

Somewhat like Chicago, the New York network has two LANs. One the private local LAN and the second is the internet LAN beyond the firewall. Unlike Chicago, we only have the one default route with no additional routes for a second LAN. This router learns all of the private subnets through BGP and uses the default route out to the internet and beyond.

One final note about this BGP configuration. You’ll notice that our three branch routers are all using the same AS number(65001). That means that they are all within the same AS and share their routing information with each other via interior BGP(iBGP). Meanwhile, the AT&T routers share routing information with ours via exterior BGP(eBGP) because they are different AS than ours and we simply peer with them. It’s not a critical distinction in this example but, the question did come up.

As we have demonstrated here, connecting to an MPLS service provider is dead simple and requires no MPLS configuration on our part. We do however have to use BGP for routing across the MPLS network, at least for this MPLS service provider(AT&T) because that is what they use. Other providers could use other routing protocols but, BGP seems to be the preferred method. But, even the BGP configuration is rather straight forward and easy to grasp.

In the next article we’ll tackle the internet router and it’s BGP configuration. It’s all very similar but, as you’ll see, there are some critical differences that may not be apparent to those new to BGP.