Saturday, December 23, 2017

NeoSD NeoGeo AES Flash Cart review

In the 00's emulation was the rough draft of a love letter from technology to classic console gaming.  Emulation made it possible to play an entire library of console games with just a PC and a modest amount of know-how.  More than that, it provided a venue for fan localization of titles that were never released in our native tongue. I will never forget the hours I spent trying to get Final Fantasy V to play on my PC, or the hours I sunk into actually playing Seiken Densetsu 3 (Secred of Mana 2).  Console emulation is praiseworthy by the fact of its existence alone.  Be that as it may, emulation has always been an approximation of the original experience.  Even with the fastest and most powerful computers, emulation doesn't seem to be able to re-create the experience with complete fidelity - often it gets close enough to scratch the itch and it's a great deal better than nothing at all, but with input lag, screen tearing glitches and a whole other litany of nitnoid issues, emulation is far from an ideal way to experience the games of yesteryear.

In the '10's that love letter to classic console gaming seems to be more fleshed out with the advent of widely available flash carts for just about every console with a cartridge slot.  These cartridges allow you to play ROM images directly on original hardware - an experience indistinguishable, in practice, from playing the original cartridge.  Usually they work by inserting an SD card with ROMs into a specially made cartridge.  When the console is powered up with the cartridge there's usually a menu where you can select a game to play from the SD card and away you go.

Not all flash carts are created equal, and the relative sophistication of specific consoles can cause compatibility issues.  In general the systems which used their cartridge slot exclusively for data storage enjoy flash carts with high compatibility, whereas other systems which used the cartridge as an extension of the hardware itself have a bit rougher of a time.  For example the SNES used several "enhancement" chips, the "SuperFX" being the most obvious one. To date there aren't any flash carts that can simulate the SuperFX (though theoretically the SD2SNES might be able to one day).  


Another example of this is the SNK NeoGeo.  The NeoGeo was primarily an arcade platform which was extended to the home market.  The arcade version is called Multi Video System (MVS) and the home version is called Advanced Entertainment System (AES) except for additional copy protection applied to the AES versions of games the systems and software are identical. This is noteworthy because SNK emphasized function over mass market economy - i.e. they did whatever it took to make the games look and feel the way they wanted even if it wasn't cost-effective.  For this reason the NeoGeo sported some of the most audibly and visually impressive games of its time, and was the most expensive console of its generation.  The original retail price of some of the games sold for the AES were more than it costs to buy a whole PS4 system these days - and that's not even accounting for currency inflation.

The NeoGeo is a massively over-engineered system. It's like if, prior to the invention of the automobile, someone decided to make horse-drawn carriages faster by strapping a hot air balloon to the top to minimize friction, attaching a foot petal mechanism for the passengers to assist driving the wheels, and harnessing a dog sled team in front of the horses for additional pulling power. Whereas most flash carts just have to mimic a single ROM chip and feed a single bus, a NeoGeo flash cart would have to mimic multiple separate chips and feed 5 buses, which requires FPGAs (basically re-configurable circuits) which are not cheap. 

For this reason a lot of people had written off the possibility of a NeoGeo flash cart ever existing.  But true to the nature of the NeoGeo itself, pragmatism has proven to be no barrier to possibility.  After years of hearing that it couldn't be done, we've got not one but two flash carts for the NeoGeo.  The NeoSD came out late 2016 and has just picked up a U.S. distributor in June of this year.  And someone named "Darksoft" on atariage.com and arcade-projects.org is nearing completion on a similar and ostensibly even better NeoGeo flash cart.


While I initially wanted to wait for the "Darksoft" cart, the old adage "a bird in the hand is worth two in the bush" won out so I recently purchased a NeoSD and wanted to share my impressions. 

The product description really doesn't do enough(for me anyway) to explain why these carts are so expensive ($400-$500) and how they differ from flash carts for other systems, so I wanted to at least share a little of what I've learned before I explain my impressions. 

I'll try to relate my experience and impressions in relatively the order I experienced them.

Getting Games
Getting games for this cart was annoying. Unlike most flash carts where ROM sets are usually available in a single archive using a common format, NeoGeo games were primarily arcade games and this cart was developed to use MAME ROMS. My attempts to just get a complete set of NeoGeo games were stymied by the fact that MAME is not packaged that way - all of the ROMs for everything are in the same giant directory and there is no way to "filter by NeoGeo" when downloading. Mildly annoying is the fact that the ROMS have to be reformatted specifically for the NeoSD. However, the NeoBuilder software which performs this function provides two additional conveniences that kind of make the whole thing a wash. Firstly the tool verifies that the ROM you have will actually work - without exception every other flash cart I've used has had at least one ROM not load because the dump was wrong, or some snowflake artist decided to write his name on the splash screen and borked the checksum. It was nice to be forewarned that a ROM was not going to work so I could replace it. But perhaps the best thing about the tool is that it compares what you have against a set of known ROMs and tells you what you're missing so you can be assured of having everything. In the end I had to go to 4 different places to get a complete set that the NeoBuilder was happy with. I would mention them here but I have a feeling that it would get censored or fall afoul of some site policy. Assuming you're like me and don't already have a complete MAME ROMS lying around, one way you could approach it is: search for "Neo Geo Set Part", get those, run them through the tool, then find a MAME torrent and download just the zip files that match the names in the "missing" column. There will still probably be one or two that don't check out properly and you can just search for those specific zip files and try what's available until you get one that NeoBuilder likes. Also - a complete set of ROMs clocks in at around 10GB so a 16GB MicroSD is more than adequate.

Packaging
This is the first time I've purchased a "Complete Edition" of a flash cart from Stone Age gamer. The overall presentation is very nice. The design is nicely conservative and the included manual turned out to be really helpful. Was a little surprised that this one did not come with a Micro SD card, but I'm guessing the margins were pretty thin. While the custom case is cool and matches the aesthetic of North American AES games, for some reason they made the case the better part of an inch taller than the standard Neo Geo cases so it won't fit on the shelf next to my other Neo Geo carts. Were these cases actually made for something else and just happened to work for AES carts? Otherwise this seems like a bizarre choice. Each cart has a serial number which is printed on a sticker. On my cart this sticker was placed on the right side (opposite the Micro SD slot) right where the cartridge inserts into the AES, so one of the letters had been marred from scraping the side of the cart slot and I couldn't tell which letter it was. I don't know if it's that way on all of the carts or just mine, but that was also kind of a bizarre choice. Otherwise the cart shell is a very nice injection-molded piece. It doesn't feel quite as substantial as the real thing but it's close.

Firmware Update
The cart I was sent was not on the latest firmware. I noticed this when the included manual mentioned options that I didn't see when the cart booted up. The manual didn't include any instructions for this, but with www.neosdstore.com plastered on the splash screen it wasn't difficult to work out. Just go to www.neosdstore.com and click the downloads link. I think the developers are very worried about clone/copycat products because they require you to create an account and register your serial number in order to get firmware updates. The firmware update file itself has the serial number in its name, and I wouldn't be surprised if the serial number is embedded in the payload as well. Anyway, the update comes with instructions and I didn't have any issues with it.

Playing Games
So, after hunting down ROMs from several sources, converting them to the required format, copying them to a Micro SD and performing a firmware update I was ready to get my NeoGeo on.

When you power on the system you're presented with a list of games and some basic options mapped to controller buttons. One thing that confused me at first was the fact that there are two separate options menus. There's a systems options menu that you get by hitting "B" (I think), and there's a game options menu that you get to by hitting "Start" with the game highlighted.

What I liked:

-The Menu
The system menu was more robust than most flash carts I've used. Each game has an image of its title screen when highlighted (I believe these are embedded in the ROM for use with MVS) as well as a nicely formatted title for the game itself (It doesn't just use the ROM file name like most flash carts). From what I've seen, Darksoft's upcoming cart is going to be more involved, but the NeoSD is already more than adequate.

-The Options
As an AES owner without a hacked console (i.e. no UNIBIOS) I've basically always been stuck with the default settings on each cartridge. Since some of these games were designed to require a ton of quarters to complete, the difficulty coupled with the inability to add more credits made it impossible for me to fully enjoy them. By setting the BIOS mode to "MVS" on the system options menu, the game options menu now allows for "Soft DIP" settings and I can set unlimited credits, adjust the difficulty maximize lives etc... This is by far my favorite feature because it has made all of my games more accessible without requiring hardware modification to my console.

What I didn't like so much:
-Load Time
When switching between games, the system will go into an "erasing/writing" phase which can sometimes take minutes. There have been video comparisons of Darksoft's cart doing this much more quickly than the NeoSD for certain games, and, in fairness, the NeoSD loading certain other games more quickly than Darksoft's. My layman's understanding of this is that not only is some kind of flash NVRAM being re-written, but certain of the FPGA circuits need to be reprogrammed as well, so this all comes down to an unavoidable consequence of the NeoGeo's complexity. While it's completely understandable, it may come as a bit of a shock just how long you're required to wait to switch games - especially newer ones like Samurai Showdown V. Once the cart is programmed, loading is instant because the NeoSD basically becomes the game that you designated until you erase it and write another game to it. If you like you can even remove the SD card and set the NeoSD to boot the game directly so it behaves exactly like an original cart.

Conclusion
It has been a blast getting re-acquainted with my AES and trying out a ton of games I never had the opportunity to play before. While it's possible to play all of these games via emulation basically for free, nothing beats the responsiveness and presentation of the real thing. My wife and I have already spent about 12 hours this weekend playing old favorites, sampling random new stuff and just having an all-around great time with this. While Darksoft's upcoming cart will arguably be better in some respects, the NeoSD has everything I want or need from a NeoGeo flash cart and I'm very happy I went with it when I did.

Monday, October 30, 2017

Seiko TV Watch

Nothing can ruin art and inspiration like money can.

When I was in 3rd or 4th grade, our science textbook at school featured a picture of the Seiko T001.

https://museum.syssrc.com/artifact/mobile/900/

I can't recall if that's the first place I laid eyes on one, but I remember becoming almost obsessed with it.  I later saw the watch featured in the movie Dragnet, and much later (though it was a much older film) in the film Octopussy.

There are plenty of sites such as the one above that tell the story of the watch much better than I can, so I'm not going to duplicate that effort.  The truth is I didn't really know the history of the watch and it didn't much matter to me either.  The fact of the matter is that in 1982 Seiko made a working television screen the size of a wristwatch face.

When I was young, the miniaturization of technology was awe inspiring.  It opened our eyes to what was possible and made an incredible promise out of the future.  It felt like every year we were closer to flying cars and wearing computers on our wrists.  Pleasure cruises to the moon were a mere generation away.

But something happened on the way to the future.  Maybe every generation looks at this progression and formulates a similar sentiment, but what the heck happened?  And the answer is simple.  Money happened.  There was no follow-up to the Seiko T001 because progressing the state of the art was not as profitable as maintaining the status quo.  Sure 30-some odd years later we have "smart" watches now - but those aren't the successor to the T001 - they're the product of an entirely different way of thinking.  The technology doesn't exist now for its own sake, it exists to keep us paying for services.  Am I being cynical?  Probably, but am I wrong?

Enough grousing about the good ol' days that never were and back to story time.

What's really weird about looking these up on Ebay is that the term "Seiko TV Watch" yields a lot of results that aren't what you're looking for.  Evidently there is a style of analog watch face that resembles the shape of an old TV set, so the vast majority of your results will be these.  Even so there is almost always a T001 up for sale on Ebay.  Don't let anyone tell you they're rare - they definitely aren't.  They're expensive because they're desirable, not scarce.

I recently had the pleasure of acquiring a Seiko T001 and it's everything I ever hoped it would be.
Buying it was actually kind of a fun experience with a couple of twists.

The whole thing consists of the watch, a pair of headphones and a receiver unit.  Most of the listings I had seen for these were missing the black foam pads from the headphones, making them look neglected and incomplete.  Occasionally the seller would offer some verbiage to the effect that the foam had disintegrated with age.  The one I bought had the foam intact in the images, and I was looking forward to getting a truly complete set.  Well, I got my foam pads alright...  Whatever material was used to make the foam back then becomes brittle over time and the vibration from being shipped caused what was left of the foam pads to basically explode in black grit all over the interior of the package.  It was pretty comical when it arrived.  I spent the first 20 minutes with the watch carefully extracting black crumbs from all its crevices.

After getting everything cleaned up, I decided to throw some batteries into it and see what it did. That's when I discovered that someone had left batteries installed into the receiver and they had predictably leaked.  Fortunately only one of the battery contacts was corroded and after a good scrubbing with some Deoxit and a toothbrush everything seemed to work just fine.

As many articles about the watch will attest, you can't really do a lot with it now because there are no more over-the-air analog TV broadcasts, and the receiver for the watch doesn't have video-in ports.  This has led a lot of people to state in the listing that it doesn't actually work anymore.  Apparently some people buy these just to collect them, and that's cool, but I bought mine to USE it. (This despite the fact that it's so wildly impractical).  So step two was figuring out how to get a video image onto the screen.

Like most portable radio receivers from the era, the headphones do double-duty as an antenna loop.  Searching around the Internet I came across a couple of accounts where people had managed to pipe a video signal into the watch by means of a cheap RF Modulator and a couple of plug adapters to convert the coaxial (type F) connector to RCA and then RCA to the phono plug used by headphones.  After fiddling around a lot with different connections, I eventually got the whole RF Modulator+adapters thing to work work by soldering a "matching transformer" (like this one https://www.amazon.com/RCA-VH54R-Matching-Transformer-VH54R/dp/B00005T3EY) inline.  I'm guessing the lower resistance of the direct connection was overpowering the tuner.  But even with that setup, the picture was very fiddly and I could never get it stable for more than a second or two.

Which led me to plan B.  Rather than wire something directly to the receiver and possibly shorten it's operational life in the process, I thought "How hard could it be to broadcast a TV signal?"   The simple answer is that it would have been a lot easier had I been trying to do it 5 to 10 years ago.  At one point in the recent past there were plentiful cheap supplies of devices like this: https://www.radiomuseum.org/r/unknown_video_sender_ut_66.html, (it's basically a box that takes an AV signal and broadcasts it very short range on a UHF channel) but I couldn't find one anywhere for love or money in the couple weeks I searched.

There are a lot of similar looking devices which only transmit between a sender and receiver and not directly to a TV, so it would be very easy to accidentally buy the wrong thing.  Several times I came across a similarly-described device and had to spend an hour trying to figure out what it actually does.  Because there are so many listing aggregators, the Internet has basically become one big echo chamber of product descriptions.  I have a feeling that the supposedly questionable legality of some of these devices led to their function being word-of-mouth only so the FCC had to guess whether or not to prohibit its sale.  Most people who knew how they worked have likely just moved on.  I did finally stumble across a compact device that sends a UHF TV signal here: https://www.ebay.com/itm/55-1190-Zealous-Analog-Audio-Video-Sender-TV-Wireless-Transmitter-Receiver/142267182316?hash=item211fc8ecec:g:n9AAAOSwA3dYlOD9 but I managed to find something a bit better (in my opinion) beforehand.

Phil's Old Radios: https://antiqueradio.org/HomeTVTransmitter.htm is an old-timey website dedicated to the preservation of old electronics and he happened to share his solution for getting a picture onto these sets after the fall of analog television broadcasts.  I have a lot of admiration for people who do this sort of thing.  Phil's website describes using a Blonder-Tongue modulator and an antenna to send a TV signal that will cover a small area.  While this is functionally equivalent and much bulkier than the all-in-one unit I saw on Ebay, it occurred to me that a piece of professional equipment was probably going to be more reliable, adjustable and repairable.  As far as I can tell the original use of these Blonder-Tongue devices was to push internal CATV channels to multiple TVs in schools and hotels, so the output is quite strong to push a signal across long cable runs.  It took some time to sort through the available listings to find the right thing and a decent price.  The original MSRP on these was nearly $2000 but they're frequently sold for between $30 and $100.  I found one for around $40.

When the modulator and antenna arrived in the mail, I assembled them and configured the modulator as well as I could work out how, got out my watch and started tuning.

After that it only took a couple of minutes fiddling with the tuner to get a solid picture on the watch.  The range was only around 20-30 feet but that was more than enough to fulfill my longtime dream of watching The Transformers on a Seiko TV watch.  It just doesn't get any more 80's than that.


Sunday, May 28, 2017

Nintendo Switch vs. Atari Lynx Which is Better

The Showdown


The Nintendo Switch has been in the hands of players for a couple of months now.  The gaming press has carried stories of shortages and stock flying off the shelves, but when you strip away the hype and marketing, how does it hold up against one of the best handhelds of all time, the Atari Lynx?

Don't let the fact that both machines feature horizontal orientation with controls on the sides fool you, these are very different devices.  We'll rank each system according to a series of 6 crucial criteria to see how they really stack up.


1. Size

Top: The Atari Lynx featuring ergonomic contouring, and ambidextrous controls. Bottom: The Nintendo Switch featuring a more stylish, but somewhat ergonomically compromised design exclusively for the right-handed. 

The Switch takes up a 9.5x4 inch surface area and is .55 inches thick, while the Lynx is 10.75x4.25 inches and 1.5 inches thick.  At first glance they are similar enough in size that it may seem like neither would enjoy a significant advantage but with a larger surface-to-screen ratio and nearly 3 times the thickness, the Lynx allows for a much sturdier and comfortable grip and more room to place your fingers while Switch players need to keep their digits much nearer the edge and pinched at a much more acute angle to grip the thinner handheld.

Advantage: Lynx


2. Erogonomics


The Lynx features a D-Pad and allows the screen to be inverted to accommodate left-handed players as well as right-handed players.  Sound volume is controlled with an analog potentiometer, allowing for an infinite range of adjustment.  The Switch also features multiple controller orientation options, but sadly none are designed with lefties in mind. The Switch tries to make up for its lack of a D-Pad with numerous buttons, analog control sticks.  The stick placement of the analog stick on the right joy-con was chosen to allow the right joy-con to operate as a standalone controller or as the right-side controls for the console when attached.  Some users find the right-analog stick placement to be an uncomfortable unwelcome compromise.  Volume control on the switch is digital, limiting the adjustment to a stair-step range of 30 different volume settings.

Advantage: Lynx


3. Screen


The Lynx sports a color 3.5 inch screen with a resolution of 160x102, while the Switch has a 6.2 inch color screen with a resolution of 1280x720.  If you do the math you'll notice pixels on the Lynx are 5x the size of pixels on the Switch!  This means more visible surface per pixel and better pixel definition.

As you can see from these direct comparison screenshots, the pixels are clearly visible on the Lynx screen, but basically impossible to see on the Switch.

With it's enormous pixel size, each individual pixel on the Lynx is clearly visible.

Shown at actual resolution, the Switch pixels are practically microscopic.


Technical details

Switch
pixel width 0.0042
pixel height 0.0042

Lynx
pixel width 0.0203
pixel height 0.0184

Advantage: Lynx

4. Power and Battery Life


The Lynx has its power socket on the top, so the unit can be placed flush on the table while it is connected to wall power.  Lynx average battery life is 4.5 hours.  The switch places its power socket on the bottom of the unit, so it cannot be placed flush on a table while connected to wall power.  Average battery life for the switch is 4.25 hours putting it nearly on-par with the Lynx, however the Lynx has changeable batteries making practical battery life infinite provided you bring enough spares.

Advantage: Lynx

5. Game Packaging

Left: Switch games come in a protective plastic clamshell for safe and easy storage. Right: Lynx games are shipped in fragile cardboard boxes which do not provide reusable protective storage.
While the game packages for both handhelds offer artwork of similar size and quality, the Switch packaging doubles as a somewhat durable and reusable storage case while Lynx owners are expected to provide an alternative means of storing games.

Advantage: Switch

5. Inside the Package - The Games

Lynx cartridges are 2.4x2.2 inches in size allowing for labels with artwork that don't require a magnifying glass to read, and they also taste pretty neutral.  While the Switch uses extremely easy to misplace cartridges 1.25 x .86 inches in size with microscopic artwork and they taste fantastically awful.  Lynx cartridges feature the pin connectors on the front side so it is very easy to tell when/if they have become dirty or need cleaning.  The pins on the Switch cartridge are hidden on the back side.  Nintendo's solution seems to have been recessing the pins to prevent surface dirt accumulation, but this also makes them extremely difficult to clean should the need arise.

Left: Switch cartridges are too small to read unassisted.  Center: Lynx Cartridges feature legible artwork and easy to inspect and clean pins.  Right: Lynx is the only one of the two to ship all of its games with helpful instruction manuals.

Also once we get into the package we discover another corner cut by the big N - switch games do not come with manuals.  In contrast every Lynx game ships with a manual.

Advantage: Lynx

6. Cartridge Selection

At present this is a no-contest category.  With it's massive selection of 74 game cartridges the Lynx completely eclipses Nintendo's plucky upstart which is still flagging behind with fewer than 10 titles available as of this writing.  It should be noted, however, that the number of titles offered for the Switch is expected to increase and may even eventually be on-par with the vast selection available to Lynx owners today.

Advantage: Lynx


Conclusion

After winning 5 out of 6 categories, it's no surprise that Atari's powerhouse comes out on top of this one.

Tuesday, April 11, 2017

Pioneer LaserActive PAC-S10 Capacitor Replacement

   The 90's were a very interesting time in the world of video games - at least they were through my rose-colored glasses.  No one had a clear picture of where the future would take us but it seemed like everyone had a picture.  This gave rise to a ton of innovation that companies these days wouldn't dare risk trying.

   One such innovation was the Pioneer LaserActive.  Basically Pioneer worked with video game companies to create a Laser Disc player with a pluggable modules that contained entire video game consoles.  At $1000 for the LD player and $600 per module, it was a contender for the crown of fiscal masochism, easily beating already prohibitively expensive enthusiast-only platforms like the SNK Neo*GEO AES.

   Pioneer managed to get both Sega and NEC onboard, so LaserActive offered both a Genesis and Turbografx-16 module.  Not only did you get the world's most expensive working Genesis or Turbografx-16 inside your Laser Disc player, plus the functionality of the CD-ROM addons offered for both consoles, you also got access to an exclusive library of games released on Laser Disc.  All of the games tried to capitalize on the playback of full motion video (which was really the only reason to use the platform in the first place) and none of them were "killer apps", but they still fetch pretty high prices because most of them are still exclusive to this specific hardware platform.  

   That's right kiddies, playing LaserActive games on an emulator is pretty much not a thing.  This is mostly due to the analog nature of Laser Disc, but more specifically how some LaserActive games used it.  Games on Laser Disc aren't exclusive to the LaserActive platform - the original Dragon's Lair arcade machines used LDs, for example.  And many if not all of the old arcade LD games can and have been emulated - hell Dragon's Lair can be played on any standard DVD player.

   Because LD's aren't digital, they can't be "ripped" in the traditional sense. The material has to be "digitized" (i.e. played back and captured in a digital format like MPEG) in order to be used by an emulator.  But with LaserActive it's not a simple matter of digitizing the analog video and re-incorporating it into the program code.  In many cases, to speed up scene transformations multiple video streams were recorded on top of one another, taking advantage of the nature of analog interlacing.  Scenes were conditionally played back as odd or even scanlines by the LaserActive software.  In other words, if you played the raw video into a capture device you'd get an indecipherable visual mess.  It's theoretically possible to either "trick" the LaserActive device into playing the properly decoded scenes into a capture device, or building a special capture device capable of decoding the specific method used to encode the video, but all things being equal, it would be extremely difficult to simply "dump" a LaserActive game, and to date I'm unaware of anyone having devoted the time and energy to attempt it.

   A couple of years ago I was able to get one of these players at a pretty reasonable price on Ebay.  Mine came with the "PAC-S10" which is the Genesis/Sega CD module.  It played "Mega LD" titles, and Sega CD titles just fine, but wouldn't recognize games in the cart slot.  While I had never heard of this particular problem, it wasn't hard to guess what it was.  The electrolytic capacitors used in electronics in the 90's are hitting an age where their failure rate is spiking.  Even the ones that weren't cheaply made or defective have reached the end of their operational lifespan.

   Any electrical engineer who would like to chime in and correct me should feel free, but from what I've been able to glean capacitors work sort of like power reservoirs.  It's my understanding that this serves a couple of purposes - firstly to "rectify" power - that is keep the amount of power moving steadily between components even when the power given off by the power supply is fluctuating.  This is particularly important when dealing with integrated circuits and data processing because the difference between a 1 and a 0 is actually measured by differences in voltage.  It may be okay if starting your washing machine in the next room makes your lights dim a fraction, but it would not be okay if your computer started mistaking 1's for 0's because of it!  Capacitors determine the amount of power that gets delivered to things like audio and video amplifiers, drive motors etc...  When they start to go bad, meaning that they're no longer delivering power at a regular expected level, all sorts of weird things start to happen.

   Electrolytic capacitors actually contain liquid electrolyte (yes, what plants crave), which is corrosive and eventually either evaporates (if you're lucky) or leaks out all over the circuit board (if you're not lucky).  The failure is based solely on age.  New in box or rarely used consoles seem to fail at the same rate as old workhorses. These ticking time bombs are unfortunately super-common because they're cheap and manufacturers don't exactly care if they only last 30 years.

   The increasing failure curve of Electrolytic capacitors has brought this problem roaring into the consciousness of classic gaming culture because most of us old guys still gaming on 30+ year old consoles are going to be affected by it sooner rather than later.  Now the common remedy to this is to replace all of the electrolytic capacitors in a given console with newer and presumably higher quality electrolytic capacitors - hopefully getting at least another 30 years out of them before they start failing all over again.

   When I bought the LaserActive it was with the knowledge that the game modules were notorious for capacitor failure, so I wasn't exactly surprised to find out that my PAC-S10 wasn't completely functional.  In fact, alongside it, I acquired a nice soldering rework station against the day I would eventually have to replace the capacitors.

   Now, the idea of replacing all of these capacitors with more that would just fail again did not sit well with me.  Fortunately not all capacitors are electrolytic.  There are quite a few different compositions.  Ceramic polymer capacitors were pretty limited and expensive in the 90's but not so much nowadays.  But would these really be compatible?

   All capacitors are rated for capacitance (imagine that!), voltage and temperature.  In this sense one is as good as another irrespective of composition.  They do have different properties, however.  For example, my quest to determine if ceramic capacitors would be a compatible substitute led me down the rabbit hole of DC Voltage Bias.  There's also the fact that most of the ceramic capacitors I found were bipolar - that is to say that polarity does not matter.  Almost all of the electrolytic capacitors in the PAC-S10 were polarized.

   Firstly, bipolar capacitors are interchangeable with polar capacitors.  In fact, you can build a bi-polar capacitor by simply connecting opposing poles (anode and cathode) of two polar capacitors with the same capacitance rating.  So there is no problem replacing a polarized capacitor with a bi-polar capacitor.

   As for the DC Voltage Bias, that's a trickier subject to tackle.  The question is will the DC Voltage Bias property of the ceramic capacitors cause issues if they're used in place of the original electrolytic capacitors.  The answer is that it depends greatly on what the capacitor is meant for.  For example, audio circuits use capacitors to regulate the frequency of signals based on a constant capacitance value.  The higher DC Voltage Bias of a ceramic capacitor causes capacitance to fluctuate when voltage is higher and can actually degrade the quality of the audio signal by altering the signal outside the desired frequency.  I'm nowhere near skilled enough to tell which circuits on the board have to do with audio with any degree of reliability, so this is a bit of a problem.  The solution seems to be relying on the voltage rating of the capacitor itself.  The DC Voltage Bias is expressed as a curve - the variation of capacitance increases exponentially as you reach the capacitor's maximum voltage value.  If that maximum value is considerably higher than the voltage is ever likely to get, the frequency variation should remain inconsequential.  By selecting a ceramic capacitor of a higher voltage value (say, double) you can minimize fluctuation by keeping the DC Voltage Bias on the low end of the curve.

   Using these assumptions I decided to go ahead with the capacitor replacement using ceramic capacitors.  I found a list of capacitors on the system and ordered the replacements from Mouser.

  Having never worked with surface-mount components before there was a bit of a learning curve.  I had never used a hot-air workstation before so I had to browse around YouTube for some pointers and examples.  The basic idea is that you blow super-hot air onto the component and the heat passes through and under the component to melt the solder.  The fact that components survive being cooked like this was somewhat amazing, but evidently they're all expected to endure that level of heat for short periods of time.  After a little experimentation I was able to get an idea of how long I needed to hold heat to a SMD capacitor before it would come loose from the board.  Just hold on the component, wave it slightly to make sure the heat is applied evenly to both sides, nudge with the tweezers every couple of seconds until it moves then pluck it off and repeat.  After the better part of 100 capacitors I went after the board with a de-soldering braid to clear off any excess solder, then used alcohol and cotton swabs to scrub melted flux and what I assume was leaked electrolyte from the board.  There was a similar learning curve with installing the replacement capacitors.  By the time I got to the end, I had to re-do the first 10 or 15 to make them look nicer.

   Replacing 90-some odd capacitors is a lot of work - especially if you're learning as you go.  By the time I put the last one in, my concentration was a little frazzled, but I was really anxious to see the fruits of my labor.  I quickly and, I thought, carefully reassembled the PAC-S10, popped it into the LaserActive player, popped a cartridge in and turned it on.  I was delighted when the screen lit up and the game actually started playing from the cartridge - that part was fixed.  But then I noticed there was no sound and my heart sank as I checked and re-checked all of the audio cables and the receiver.  As I sat down to think through what might have gone wrong, I realized I had no memory of re-attaching the daughter board when I reassembled the PAC-S10.  I went back to my work area, and sure enough, the daughter board was still sitting there waiting patiently for me to figure out what had gone wrong.  With the unit _completely_ re-assembled the sound was back.  I played through about 3 stages of Lightning Force, 2 stages of The Terminator on Sega CD, and 2 stages of Hyperion on MegaLD before I was satisfied that everything was working.

   A couple of weeks later I was playing around with Laserdiscs and decided to pop in one of the Star Wars Definitive Edition discs and I noticed a low buzzing noise when the sound was turned up.  After poking around I noticed that the buzz went away when the Laserdisc was played with the PAC module ejected, so the buzz was definitely coming from it.  Problem is I didn't really do any before and after testing so I don't really have any way of knowing if this was a result of using ceramic capacitors, or if it was just poor shielding of the PAC module causing some minor noise.  No way I'm replacing all of the capacitors again to find out!

   

Monday, April 10, 2017

Zelda

The somber first note of the title theme plays
in my head every time I see this image.
Even after completing the main quests, I find myself being drawn back to Breath of the Wild.  This is most likely because the Switch is new and shiny and I haven't been limited by such a lack of options since I was a kid.  But I also have to admit that this is by far the most satisfying Zelda experience I've ever had.  Zelda has always been a mixed bag for me.

Nintendo in-store demo kiosk from the 80's.
As a kid I remember Zelda being an esteemed phenomenon.  Whenever my mother took me with her while she shopped at the Richman Gordman department store in Council Bluffs, Iowa, I would immediately head left from the entrance to the electronics counter where the Nintendo demo kiosk sat.  Most of the time there was already someone playing and a line of kids waiting for a turn.  The shiny metallic gold of the Zelda cartridge practically leaped out at the eyes. The first time I ever played the Legend of Zelda was on that demo kiosk.  Play time was limited to something like 5 minutes so I only ever got to see the opening and play for a couple of minutes before the game reset, but that hauntingly adventurous title theme soaked into my blood.


The actual Richman Gordman where I first played the Legend
of Zelda on a demo kiosk.  This photo is from it's Grand
Opening about 12 years beforehand.  It was vacated about
a decade ago, and torn down in the last year or so.
The game was always rented out whenever we went to the video store (I just realized how much of an anachronism "video store" has become.).  One of the teachers at school was a huge fan too.  There was a group of boys he would discuss the game with.  I usually sat at a distance and just listened to the conversation wondering what the actual experience must be like.  By the time I actually did get to rent the game and sit down for more than 5 minutes to play it, I was completely confused.  I wandered around aimlessly for about 30 minutes, stabbing stuff, dying and repeating.  At one point I found a dungeon, but never figured out how to solve it.  I didn't have any of the maps or guides or network of friends to ask for advice.  The game didn't do anything to explain itself, really.  After a while I grew frustrated and went off to go seek less demanding sources of fun.  Oddly, my failure to get anything out of it did very little to diminish the mystique - if anything Zelda became even more mythical in my mind as a result.

To a certain degree most of my experiences with Zelda have followed the same pattern.  When Zelda II came out, I rented it, stabbed around for an hour or two without really understanding what was going on, then shut it off and went outside with a bucket and my bike to go hunt for snakes.

Pre-production images from The Legend of Zelda:
A Link to the Past.  This was taken from a poster that
was widely distributed with new games.
When the new Zelda title was announced for the SNES and the first screenshots started to trickle out, I was instantly enraptured by it.  The leap in visual fidelity over the NES games brought the artistry close enough to the surface of reality to fuel my imagination and fill me with a sense of adventure I've seldom experienced since.  However, by the time A Link to the Past finally came out, life had intervened and I no longer had a SNES, or any video games of any kind.  It wasn't until about 1995 when someone gave me a SNES in lieu of repaying a debt that I actually started my collection as it is today.  By that point SNES games were plentiful and cheap and there were a dozen places to buy them from.  I went to a local shop called Comic City and bought a handful of SNES games - among them A Link to the Past.

A Link to the Past was every bit the experience I wanted it to be.  Adventurous music, the joy of discovery, and for the first time, the game had clearly defined objectives.  You were free to wander around in search of new secrets but the game had built-in maps, and objective markers. There was also a fortuneteller that functioned as a source of hints about how to progress the main story.  I didn't need a manual or a strategy guide to play or complete it.

After finishing LttP I hunted down a Super Gameboy ( a peripheral that allowed you to play Gameboy games on the SNES ) and a copy of Link's Awakening.  Link's Gameboy outing was a bit smaller in scope than LttP, but it was every bit as fun.  There was only one time I ended up stuck because the game assumed I had a manual and had read it. At that point, every Zelda game I had played had been used and none of them had come with manuals.  There's a particular dungeon with a locked room that can't be opened until you decipher the clue: "First, defeat the imprisoned Pols Voice, Last Stalfos."

The infuriating puzzle from Link's Awakening that I struggled with because I didn't have a manual.
After over 20 years I still remember that clue because I mulled it so long and so hard.  I actually called Nintendo and purchased a strategy guide for the game ( Nintendo Power - Nintendo's in-house magazine - had a perpetual promotion during the 90's where you would get a free official strategy guide for whatever game you liked whenever you subscribed or renewed your subscription - I renewed my subscription to get the guide.)  To my utter frustration the strategy guide also assumed I had a manual and infuriatingly just repeated the re-worded clue "Defeat the pols voice first then stafflos."  I ended up solving the puzzle randomly out of sheer perseverance.  It wasn't until the door finally unlocked and I began to analyze what I had done that I realized what I had been missing.  It was not at all clear to me before that "pols voice" was the name of the little rabbit-face enemies, and that stalfos were the skeletal enemies.  The trick was simply to kill the baddies in the correct order.

Ocarina of Time was the first Zelda game I actually got to participate in on day one.  Nintendo did more promotion for that game than I've seen before or since.  Every retailer had some kind of pre-order bonus, and I made it my goal to get as many as possible.  I ended up pre-ording the game at something like 5 different places (though I only bought one copy) - I even went so far as to drive 200 miles to Kansas City to hit a GameCrazy.  There were the usual posters and T-shirts, but also mini figures, stickers, hats, keychains, phone cards, and even a Trapper-Keeper-like binder.  I also managed to get an 8-foot-tall sword standee from Wal*Mart. On release day, I got a call from the manager at Electronics Boutique telling me that I better get there quickly because they did not receive enough stock to fill all pre-orders.  I told my boss that I had to take off work for a couple of hours to run an errand.  I tried very hard to let her get the wrong impression - like this was some kind of family emergency, but when she asked me directly I had to tell her.  She was normally pretty strict and I fully expected her to deny my request when I told her I was driving across town for a video game.  Evidently her grandson had been talking about the new Zelda non-stop so she let me go with a smile, and when I got back to work, she asked if she could look at the box and remarked how "neat" it was.  The rest of the day I had that shiny gold box sitting on my desk reminding me that I was not at home playing.

Pictures I took on November 21st 1998 showing off a good portion of my pre-order swag as well as magazines featuring Ocarina of Time on the cover and, of course, the rest of my Zelda collection at the time.

When I finally did get home, my wife and I both gathered around to watch the opening scenes together.  As awesome as it was playing a polygonal Zelda game for the first time, that familiar feeling of being lost set in.  The game attempted to explain itself, but stopped just short of telling you exactly what to do in a lot of circumstances.  I had manuals and strategy guides (at least two of them, in fact - I collected _everything_ OoT), but the experience was never as fun as LttP and I eventually put it down before completing it and moved on to other things.  It wasn't until 2015 when I finished setting up my retro gaming area that I pulled out my original OoT cart and completed the game.

Prior to Ocarina of Time, Zelda releases were rare and special events.  Nintendo was very careful and deliberate about every release and they took years to ensure they were great.  Between 1986 and 1998 (12 years), there had been a total of 5 Zelda releases.  There were another 5 Zelda titles released between 2000 and 2002.

The first of the rapid-fire Zelda releases was Majora's Mask which sought to re-use the game engine and assets from OoT but inject a bunch of new gameplay ideas to avoid just repeating the same convention.  Problem was I had never grown comfortable with the convention they sought to buck, so layering more complexity over top of it kind of turned me off to Majora's Mask.  I bought it, played it, and put it down.

Next up was Oracle of Seasons/Oracle of Ages.  After Pokemon's runaway success, there were a lot of other Gameboy releases that tried to mimic the convention of simultaneously releasing two nearly identical games with different content.  Oracle of Seasons/Ages provided two drastically different quests.  I bought these, but never really got into playing them.  They were developed out-of-house by Capcom, so I was a little leery but I think I was just experiencing Zelda overload at the time.

Although it is often cited as its own release, Four Swords was more of a mini-game that was attached to the Gameboy Advance remaster of A Link to the Past.  I was not at all interested in playing it because of the glut of shoehorned multiplayer going around at the time.

When Nintendo unveiled Wind Waker, I nearly declared the series dead.  All those years waiting for a photo-realistic Zelda experience, Nintendo finally produced hardware that could pull it off and instead they were going to give us an impressionistic cartoon.  My wife peeled me off the ceiling and asked if ANY Zelda game had ever turned out to be awful and I had to admit, that no, they hadn't.  I warmed to the idea and my enthusiasm grew so great that I ended up importing the Japanese version of the game.  Wind Waker turned out to be an amazing and magical experience that renewed my love for the series.  It brought back the music and discovery - the sense of exploration, and most importantly the game just seemed to click with the way I thought.  I was able to intuit where treasures would be and how to solve certain puzzles.  I finished the game in Japanese before the U.S. version was released.

The Minish Cap was the first original Zelda title to come out for the Gameboy Advance.  It's art style was more cartoonish than previous titles, but not quite the same as Wind Waker.  I remember it being a fun game, but I found the gimmick of a hat that shrunk Link to be uninteresting and never really finished it.

Twilight Princess came at a really odd time.  It was developed for the Gamecube but ported to the Wii as a launch title.  It was simultaneously released for both consoles with the Wii version incorporating motion controls. Up until this point Link was generally left-handed and the environments were apparently designed with his left-handedness in mind.  Since the overwhelming majority of players would be using the Wii motion controls with a dominant right-hand, they made Link right-handed in the Wii version of Twilight Princess.  This also caused them to horizontally flip the entire game.

I remember waiting in line in front of Best Buy all night long in the freezing cold the day before the Wii launch.  I read half of Eragon shuddering under a blanket.  I was the second or third in line when the store opened.  Grabbed up my Wii, copy of Twilight Princess, and Red Steel, a Pro Controller, extra Wii-mote and Nunchuck and headed home to connect the system, play for 10 minutes and collapse from exhaustion.  Twilight Princess was a pretty game - the more realistic Zelda game everyone had been clamoring for, but it failed to really capture my imagination.  We were back to a system that felt unintuitive to me and I was often wandering around lost.  After playing the game for about 10 hours or so, I put it down and didn't touch it for years.  I did eventually finish it about 5 years after it came out.

When the Phantom Hourglass came out for the DS, I was pretty excited when I saw it shared its art style with Wind Waker.  What could be better than Wind Waker on my DS?  Come to find out, in a bid to force players to use a new control scheme, Nintendo had done away with traditional D-Pad movement and required players to move Link around the environment with a stylus.  This meant that I had to block my view of the action to move Link around.  After 5 minutes I quit out of disgust and never touched it again. If I wanted to use a stylus I'd play games on a PDA. Such a waste.

The next game in the series - Spirit Tracks - was also sacrificed to the stylus gimmick.  I bought a copy out of a bargain bin for $5 to keep my collection complete, but have never played it.

Skyward Sword was a game that felt like it was crippled by out-of-date hardware and gimmicky control schemes.  While not as egregious as the Phantom Hourglass's forced stylus control, it was nevertheless annoying. The game was designed to use the new Wii Motion Plus - which was basically an iterative improvement in accuracy for the Wii Remote.  The player would swing the controller like a sword hilt to attack, or to activate special actions.  It worked well enough, but it was tiring and eventually got in the way of the gameplay.  But probably the most disappointing aspect of Skyward Sword was the fact that it was released for the Wii.  The Wii launched in 2006 during the transition to HDTV, and to keep costs low Nintendo chose hardware that was locked to a comparatively low resolution ( 480p ).  This wasn't such a big deal at the beginning of the Wii's lifespan, but 5 years later the Wii's graphics were downright ugly on contemporary TVs, so the artistry and beauty of the game was forever marred by low resolution graphics.  I can't bring myself to try to finish this game in that state - hopefully it will be redeemed with an HD remaster some day.

Click to Enlarge
Click on the image to see the full effect.  This side-by-side was taken using an emulator.
Both images are of the exact same game code and textures, but on the left you see what the game
 looks like trapped behind the Wii's low resolution 480p output and on the right you see 
what's possible when the unaltered game is simply rendered in a modern HD resolution. 

The Wii U was the first console Nintendo released with the capability of rendering in HD (720p/1080p).  The console was home to HD remakes of Wind Waker and Twilight Princess, which brought a similar graphical improvement to the images of Skyward Sword above.

Breath of the Wild is, by far, the largest departure from the basic Zelda formula.  It more closely resembles western-developed RPGs like the Elder Scrolls, Assassin's Creed, or Far Cry.  It uses the same basic method of quest organization and branching, the same method of mapping ( including map progression ).  It uses the same core control scheme as every third person game released in the last decade - no forced experimentation.  Yet despite all this it manages to retain the soul of a Zelda game with massive amounts of exploration and discovery, skill and tool-based progression, and clever puzzles and secrets.

Wednesday, March 08, 2017

Nintendo Switch Launch - Or Why I Was Happy To Spend 14 Hours In The Freezing Cold

     Nintendo had been advertising for months that on January 12, 2017 we'd finally start getting some details about the new platform they'd been working on as a successor to the Wii U - codenamed "NX".  Hopes and hype were high, and, of course, most of the important details leaked in the days leading up to the reveal.  We learned that the console was a portable/home console hybrid and that its official name would be "Switch".  It was a closely guarded secret what day it would launch and what its price would be.

     The big Switch reveal was broadcast worldwide, so in order to allow maximum viewership, it was done late that night, ending just prior to midnight CST.  The Switch would launch on March 3rd and cost $300.

     No game system is really worth anything without quality games, and Nintendo made sure one of the launch titles was from the Zelda series - one of their two biggest properties and practically guaranteed to be amazing.  The new Zelda game, "Breath of the Wild" had been in development for years on their previous platform,  the Wii U.  I mention this because I already had a standing pre-order for Zelda on the Wii U from before we knew the release date or even the name.

     Even if you're not a gamer you probably know that nowadays that when a new console comes out the only way to be sure to get one day one is to preorder or wait in line the day of.

     After staying up another hour waiting for pre-orders to go live, Nintendo made a Twitter announcement that made it seem like pre-orders wouldn't be available until after 10 AM the next morning.  This was great, I thought, because it would give everyone a fair shot at getting one in.

     So about 1:15 AM, I set my alarm and turned in.  In the morning I awoke to the fact that the pre-orders had gone live just minutes after I went to bed and were mostly gone.  In the night news had also dropped about two premium versions of the new Zelda game - a Special Edition and an even bigger Master Edition.  These, of course, had sold out faster than the consoles.  This was a heart-crushing blow because Zelda is pretty central to my game collection, and I absolutely love big limited editions like that.  What was more infuriating was the fact that a sizable percentage of the disgusting opportunists who had consumed all of the Master Edition pre-orders had done so for the shameful purpose of re-selling them at obscene prices on Ebay.  They were all over the site with 400-500% markups within minutes of the preorders going live.

     I was able to get a pre-order in for the system online at Target, but no such luck with the premium Zelda games.  After realizing I was pretty much out of luck pre-ordering online, I got dressed and headed straight for Gamestop.  They sometimes hold back stock of rare items like that for in-store pre-sales (where you pay full price and have the item delivered to your home) so if you get there early enough you can get an order in even if they're sold out online.  When I arrived I was first in line but after an hour wait for them to open, the manager informed me they were not taking any in-store orders of any kind for the Zelda Master Edition.  I pre-ordered the Special Edition and left.  At least I'd be able to get that one.

     Of course I wasn't going to give up that easily.  We have an independent game shop in town that specializes in rare releases like this - they have almost all of the Limited Run Games releases, and I even saw a FFXV Ulitimate Edition in their case once (this is noteworthy because that particular game was extremely limited and not sold at retail).  I called and inquired and the manager/owner wasn't quite sure if they were going to get any Zelda Master Editions, but he asked me to call back the next day.  I called back the next day and he informed me that they would be getting about 3 of them and I could come down and pre-order it in person.  One car ride later I had my pre-order ticket and all was right with the world.

     A couple of days later another new Switch game - Super Bomberman R went up for pre-order.  It wasn't limited or premium so there was no mad rush to get it.  I pre-ordered it from Amazon along with a screen protector confident in the knowledge that Amazon had their stuff together and wouldn't let me down.

     Then things took a downturn. A couple weeks before launch I received a call.  The guy at the local brick-and-mortar was informed by Nintendo that he was not worthy of receiving inventory of the Master Edition after all.  Without more information I can only speculate on what exactly went down.  The guy was super apologetic and told me that if the situation changed and he ended up with inventory anyway that I'd get first dibs, but my pre-order was no longer worth the paper it was printed on (well I got store credit for what I put down, but that wouldn't help me get a Master Edition).  At that point I wanted to give up on the idea of getting a Master Edition because the thought of feeding one of those reprobate scalpers was worse than not having it.

     So what could I do?  Pray.  I asked God for help.  Why would God help me get an expensive toy when there are people dying of cancer and starving on the other side of the world?  Not because I deserve it.  Maybe because I was humble enough to ask?  I can't say with certainty.  But what I can say is that it's never a waste of time to talk to God.  I'd wager that most of what we think are moral and worthy causes are equally as meaningless as expensive toys from His perspective.  The fact that he helps any of us at all with anything is frankly amazing.

     The next blow came from Target about a week later.  My console would not be delivered on launch date, but AFTER the weekend when I didn't have time to enjoy it properly.  I called twice to see if I could upgrade to overnight shipping.  After a stilted and worse than usual English-as-a-second-language conversation with Target's customer support,  I finally worked out that once the order was placed it was immediately etched in stone and could not be altered.

     It was shaping up to be a pretty dismal launch - I wouldn't get the new system until after the weekend, and I wouldn't get the Master Edition.  But at least I could count on Amazon.  Unlike Target they have this whole thing worked out so 99% of new games are delivered on launch day if you pre-order.  I was sure to get Super Bomberman R, and if I was desperate for Zelda I could play it on the Wii U.


     When March 1 rolled around, I was informed that even the juggernaut Amazon was going to let me down.  My orders for the Wii U version of Zelda and Bomberman R were both now in limbo and definitely wouldn't be delivered until after the launch weekend - if ever.

     By March 2nd I was pretty bummed and expected that this would be the first Nintendo console in 21 years that I didn't get to have on day 1.  And then something miraculous happened.  An email from Best Buy popped into my inbox explaining that they were having midnight openings in selected locations.  Not a big deal on the surface of it, and not unexpected.  There was a store in my city that was holding one of the openings and the listing said that there would be a limited supply of consoles for non-pre-order customers.  I figured it would be worth spending a couple of hours in line to get the console on day 1.  I could always return the one I pre-ordered from Target if I managed to get one at midnight.

     When Best Buy opened I called and asked for a little detail about how they were handling the midnight opening, and fully expecting the answer to be "no", I asked if they would have any Zelda Master Editions that weren't already spoken for.  To my absolute shock he answered that there were two stores in the area that should have at least one, but of course they would only be available on a first-come first-served basis.  Also, neither of the stores that would have the Master Edition were having a midnight opening.  To verify with absolute certainty that they indeed had a Master Edition available, he told me to call back that evening and speak with his night time counterpart to visually verify that the store had received one.  I started to allow myself to hope a little and my mind raced with plans.

Only 14 hours to go!
    When I got home from work that day, I forced myself to take a nap for a couple of hours, woke up and called the store again at 7PM.  The guy who answered told me they did indeed have a Master Edition that would be put out for sale and described it to me in detail so I was sure we were both talking about THE Master Edition.  I asked him where it would be when the store opened so I knew exactly where to find it.  He was very helpful, but there was too much at stake to leave this up to chance, so I decided to visit the store that evening before they closed.  I put on 2-5 layers of clothing, packed my sleeping bag, some snacks, some drinks, and an ebook reader and headed out.  When I arrived at the store, I couldn't find the gentleman I had spoken with on the phone, but was able to get the manager to confirm for me that they had at least one Master Edition - she also described it in detail and followed up with a very thorough explanation that it couldn't be held, reserved or otherwise spoken for.  They would be passing out tickets for the consoles themselves in the morning but not for anything else i.e. not for the Master Edition.  But IT WAS THERE!!!! Originally I had planned to maybe hit the midnight opening at the other Best Buy and head to this one after, but the possibility of someone beating me to that Master Edition made me resolve then and there to plant myself in front of the door and wait.  I was just going to be spending the entire interval obsessing over it anyway - might as well do something useful with that energy. That was just before 8PM.  The store would not open again until 10AM.

     So I rushed to the car, pulled out my supplies and set up right in front of the entrance for a 14-hour wait to save both the integrity of my Zelda collection, and my 21-year streak of having every new Nintendo platform on day 1.

     I'm no stranger to waiting in line for stuff like this.  I did it for the PS2, the DS, the PSP, the Wii, and the PS3.  On every one of those occasions there were always people driving by staring.  Many would roll the window down and ask what we were waiting for, and in almost every instance, someone would make fun of us for waiting in line or call us crazy.  This time my experience was a lot different.  I had set up about an hour before the store closed for the night so around a dozen customers came and went in that time.  The ones that bothered to acknowledge my presence would pause, cock their head and then say something like "Oh, he's going to get a Switch." And to my surprise it was almost always followed by something complimentary like "That's dedication." or "He's gonna get one."  As the Best Buy employees left, they would sometimes stop to make sure I knew that the store wasn't opening at midnight, or to tell me they were impressed with my determination.  The manager I had spoken with earlier left at around 10:30PM and took the time to tell me cheerfully that she'd see me in the morning, and asked if I was going to be warm enough.  I was impressed by the level of awareness and acceptance of everyone who stopped by.  Waiting in line for over a dozen hours for a toy was no longer a fringe activity, it was accepted and even lauded.  Some things really do change.

     There were a couple of people that stopped by around 11 to get in line with me but they were completely bewildered when I told them that this particular Best Buy was not opening at Midnight and left shortly afterward.  It wasn't until around midnight that the next guy crazy enough to wait all night showed up.  He was a night-shifter so he didn't get off of work until 11PM and knew that was way too late to get in line for a midnight opening.  He was woefully unprepared for how cold it was outside.  Truthfully I was still freezing underneath all of those layers.  20 degrees to 98.6 is one heck of a differential.  We chatted for a hour or so.  I offered to hold his spot in case anyone else showed up so he could use the restroom at a nearby gas station.  He returned about 30 minutes later and handed me a packet of chemical hand warmers (man was that a kingly gift).

     My wife would text me occasionally - she had graciously offered to stop by and hold my spot while I used the restroom and/or warmed up in the car.  Around 3AM I took her up on it.  After a brief bathroom break I was right back at it.  Shortly after that the other guy in line left to go somewhere and sleep, leaving the seat from his van next to me as a placeholder.  He came back around 6AM and I was still the only person in line.  Around 7 my wife came back to give me another bathroom break.  It wasn't until around 8 that others began to gather around and wait in line.  My wife came and carried away the chair and sleeping bag around 9AM so I could stand for the last hour of waiting - and the last hour is ALWAYS the longest...

Safely Nestled in My Car - the Spoils of Victory
     As promised, a manager came out at 9:30AM to hand out tickets for consoles.  People wandered back to their cars for warmth, but, of course, I wouldn't budge from the front of the line.  When the doors finally opened at 10, I rushed inside to the spot where the manager had told me it would be, and there it was - the Master Edition, the one and only Master edition in all it's glory.  The manager was standing beside it ready to guide me to it on the off chance that my eyes hadn't been drawn to it like rare earth magnets to a refrigerator.  I had been unable to talk them out of putting one of those terrible anti-theft devices on it so there were little indents along some of the edges, but after what I went through, those are battle scars!  I snatched it up, and heard to the disappointed voices of the people behind me asking if there were any others.  I rushed it over to the counter and begged the clerk to immediately remove the anti-theft device before it could do any more damage.  I didn't even bother to pick up my console until the Master Edition was safely in my car.  After tucking it gently into my trunk, I headed back inside to redeem my coupon for a Switch. While inside, one of the other managers spotted me and since I wasn't carrying it anymore asked almost in concern approaching panic if I had gotten my Master Edition. She didn't realize I had left and come back already. Don't think it didn't enter into my thoughts that someone might have been desperate enough to break into my car to get that Master Edition!

     It was done - launch day had been saved! I ended up with the Master Edition and a console on day 1. Thank the maker.

     The epilogue to my story was that on the way back home, I swung by Gamestop to pick up my Special Edition of Zelda (the less-mega-rare premium version) and subsequently spent the weekend absorbed in the new Zelda game and really enjoying the new console.

     Amazon had botched the Switch launch so badly that I now count it as a blessing that I didn't manage to get a pre-order in with them.  They actually ended up losing my Super Bomberman R order altogether.  When Monday came, my pre-ordered console from Target arrived and I promptly sold it (at cost) to a friend who wasn't able to get one.

I love happy endings:)


     

Monday, February 06, 2017

Unmounting a volume with open file handles to deleted files without killing the process

Recently we've been turning on compression in our Oracle databases because it saves us a ton of disk space and actually improves the performance of the databases.  Among other things the process involves writing the newly compressed tablespaces to new data files (the systems in question are not using ASM).  Because we're moving the DB's from where they are, that gives us the opportunity to do some housecleaning before moving them back, such as un-mounting, checking and possibly resizing the existing filesystems.

Occasionally it will occur that after the data has been migrated away and the old data files deleted, that Oracle still has open handles to the deleted files.

So the DBA tells me the filesystem is clear and I can have it, but then this happens:

[root@kwt-r3oql00 E1Q]# umount /dev/mapper/vg_kwt_r3oql20_s00-oracle_E1Q_sapdata7
umount: /oracle/E1Q/sapdata7: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))

And when we go to see who the culprit is:

[root@kwt-r3oql00 E1Q]# lsof /oracle/E1Q/sapdata7
COMMAND     PID   USER   FD   TYPE DEVICE    SIZE/OFF     NODE NAME
oracle_13 13214 oracle  520u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_13 13254 oracle  519u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_13 13258 oracle  520u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_13 13286 oracle  517u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_13 13294 oracle  520u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_13 13526 oracle  506u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_13 13996 oracle  512u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_14 14424 oracle  493u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_14 14488 oracle  515u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_14 14574 oracle  403u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_14 14712 oracle  506u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_23 23296 oracle  273u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_24 24260 oracle  277u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)


This is super annoying because the DB seems to have forgotten about those open handles and internally has no mechanism for gracefully releasing them.  If you can tolerate the outage restarting the DB will release those handles, but if you can't there is another solution.

The GNU debugger (gdb) will allow you to attach to the process and close the file handles.  It's not without risk, but I've been using it successfully for years.  The main drawback is how tedious it is to sift through /proc looking for the open file handles and closing them one by one with the debugger interactively.

Laziness being the father of invention, I wrote the following script to do this for you.  Simply provide it the path to check for open deleted file handles, and it will find the processes, attach the debugger and remove the handles.


#!/bin/bash

FS=$1

if [[ -z $FS ]]; then
   echo "Please provide a filesystem path to check for open file descriptors to deleted files"
   exit 1
fi

echo "WARNING: This is super-dangerous.  Please don't use it in Prod without a "
echo "         really good reason and a change request/blackout"
read -p "Type 'C' and Enter to continue, anything else to abort: " CHECK

if [[ "$CHECK" != "C" ]]; then
   echo "Aborted by user.  No changes made."
   exit 0
fi

# Get a list of processes with open file handles to the given directory
PIDLIST=`lsof $FS | egrep -v 'PID' | awk '{print $2}' | sort -u`


for PID in $PIDLIST; do

  unset DESCLIST

  # Get a list of file descriptors for that PID that refer to deleted files
  DESCLIST=`ls -l /proc/${PID}/fd | grep deleted | awk '{print $9}'`

  # Display the list
  echo "The Process $PID has open file descriptors for deleted files:"
  echo "${DESCLIST}" | sed 's/^/  /g'

  # Create a name for the script for gdb
  DEBUGSCRIPT=/tmp/.$PID

  # Remove any previous version of the script
  if [[ -f $DEBUGSCRIPT ]]; then /bin/rm $DEBUGSCRIPT; fi

  # Write a close command for each deleted file descriptor
  for DFD in $DESCLIST; do
     echo "p close(${DFD})" >> $DEBUGSCRIPT
  done

  # Detach and close the debugger
  echo "detach" >> $DEBUGSCRIPT
  echo "quit" >> $DEBUGSCRIPT

  echo "Forcibly closing handles for deleted files on process $PID"

  # Run the debugger in batch mode to execute the script and close the handles
  /usr/bin/gdb --pid $PID --batch -x $DEBUGSCRIPT

  # Wait before deleting the script
  sleep 1

  # Clean up the script
  /bin/rm $DEBUGSCRIPT

done