Wednesday, March 08, 2017

Nintendo Switch Launch - Or Why I Was Happy To Spend 14 Hours In The Freezing Cold

     Nintendo had been advertising for months that on January 12, 2017 we'd finally start getting some details about the new platform they'd been working on as a successor to the Wii U - codenamed "NX".  Hopes and hype were high, and, of course, most of the important details leaked in the days leading up to the reveal.  We learned that the console was a portable/home console hybrid and that its official name would be "Switch".  It was a closely guarded secret what day it would launch and what its price would be.

     The big Switch reveal was broadcast worldwide, so in order to allow maximum viewership, it was done late that night, ending just prior to midnight CST.  The Switch would launch on March 3rd and cost $300.

     No game system is really worth anything without quality games, and Nintendo made sure one of the launch titles was from the Zelda series - one of their two biggest properties and practically guaranteed to be amazing.  The new Zelda game, "Breath of the Wild" had been in development for years on their previous platform,  the Wii U.  I mention this because I already had a standing pre-order for Zelda on the Wii U from before we knew the release date or even the name.

     Even if you're not a gamer you probably know that nowadays that when a new console comes out the only way to be sure to get one day one is to preorder or wait in line the day of.

     After staying up another hour waiting for pre-orders to go live, Nintendo made a Twitter announcement that made it seem like pre-orders wouldn't be available until after 10 AM the next morning.  This was great, I thought, because it would give everyone a fair shot at getting one in.

     So about 1:15 AM, I set my alarm and turned in.  In the morning I awoke to the fact that the pre-orders had gone live just minutes after I went to bed and were mostly gone.  In the night news had also dropped about two premium versions of the new Zelda game - a Special Edition and an even bigger Master Edition.  These, of course, had sold out faster than the consoles.  This was a heart-crushing blow because Zelda is pretty central to my game collection, and I absolutely love big limited editions like that.  What was more infuriating was the fact that a sizable percentage of the disgusting opportunists who had consumed all of the Master Edition pre-orders had done so for the shameful purpose of re-selling them at obscene prices on Ebay.  They were all over the site with 400-500% markups within minutes of the preorders going live.

     I was able to get a pre-order in for the system online at Target, but no such luck with the premium Zelda games.  After realizing I was pretty much out of luck pre-ordering online, I got dressed and headed straight for Gamestop.  They sometimes hold back stock of rare items like that for in-store pre-sales (where you pay full price and have the item delivered to your home) so if you get there early enough you can get an order in even if they're sold out online.  When I arrived I was first in line but after an hour wait for them to open, the manager informed me they were not taking any in-store orders of any kind for the Zelda Master Edition.  I pre-ordered the Special Edition and left.  At least I'd be able to get that one.

     Of course I wasn't going to give up that easily.  We have an independent game shop in town that specializes in rare releases like this - they have almost all of the Limited Run Games releases, and I even saw a FFXV Ulitimate Edition in their case once (this is noteworthy because that particular game was extremely limited and not sold at retail).  I called and inquired and the manager/owner wasn't quite sure if they were going to get any Zelda Master Editions, but he asked me to call back the next day.  I called back the next day and he informed me that they would be getting about 3 of them and I could come down and pre-order it in person.  One car ride later I had my pre-order ticket and all was right with the world.

     A couple of days later another new Switch game - Super Bomberman R went up for pre-order.  It wasn't limited or premium so there was no mad rush to get it.  I pre-ordered it from Amazon along with a screen protector confident in the knowledge that Amazon had their stuff together and wouldn't let me down.

     Then things took a downturn. A couple weeks before launch I received a call.  The guy at the local brick-and-mortar was informed by Nintendo that he was not worthy of receiving inventory of the Master Edition after all.  Without more information I can only speculate on what exactly went down.  The guy was super apologetic and told me that if the situation changed and he ended up with inventory anyway that I'd get first dibs, but my pre-order was no longer worth the paper it was printed on (well I got store credit for what I put down, but that wouldn't help me get a Master Edition).  At that point I wanted to give up on the idea of getting a Master Edition because the thought of feeding one of those reprobate scalpers was worse than not having it.

     So what could I do?  Pray.  I asked God for help.  Why would God help me get an expensive toy when there are people dying of cancer and starving on the other side of the world?  Not because I deserve it.  Maybe because I was humble enough to ask?  I can't say with certainty.  But what I can say is that it's never a waste of time to talk to God.  I'd wager that most of what we think are moral and worthy causes are equally as meaningless as expensive toys from His perspective.  The fact that he helps any of us at all with anything is frankly amazing.

     The next blow came from Target about a week later.  My console would not be delivered on launch date, but AFTER the weekend when I didn't have time to enjoy it properly.  I called twice to see if I could upgrade to overnight shipping.  After a stilted and worse than usual English-as-a-second-language conversation with Target's customer support,  I finally worked out that once the order was placed it was immediately etched in stone and could not be altered.

     It was shaping up to be a pretty dismal launch - I wouldn't get the new system until after the weekend, and I wouldn't get the Master Edition.  But at least I could count on Amazon.  Unlike Target they have this whole thing worked out so 99% of new games are delivered on launch day if you pre-order.  I was sure to get Super Bomberman R, and if I was desperate for Zelda I could play it on the Wii U.


     When March 1 rolled around, I was informed that even the juggernaut Amazon was going to let me down.  My orders for the Wii U version of Zelda and Bomberman R were both now in limbo and definitely wouldn't be delivered until after the launch weekend - if ever.

     By March 2nd I was pretty bummed and expected that this would be the first Nintendo console in 21 years that I didn't get to have on day 1.  And then something miraculous happened.  An email from Best Buy popped into my inbox explaining that they were having midnight openings in selected locations.  Not a big deal on the surface of it, and not unexpected.  There was a store in my city that was holding one of the openings and the listing said that there would be a limited supply of consoles for non-pre-order customers.  I figured it would be worth spending a couple of hours in line to get the console on day 1.  I could always return the one I pre-ordered from Target if I managed to get one at midnight.

     When Best Buy opened I called and asked for a little detail about how they were handling the midnight opening, and fully expecting the answer to be "no", I asked if they would have any Zelda Master Editions that weren't already spoken for.  To my absolute shock he answered that there were two stores in the area that should have at least one, but of course they would only be available on a first-come first-served basis.  Also, neither of the stores that would have the Master Edition were having a midnight opening.  To verify with absolute certainty that they indeed had a Master Edition available, he told me to call back that evening and speak with his night time counterpart to visually verify that the store had received one.  I started to allow myself to hope a little and my mind raced with plans.

Only 14 hours to go!
    When I got home from work that day, I forced myself to take a nap for a couple of hours, woke up and called the store again at 7PM.  The guy who answered told me they did indeed have a Master Edition that would be put out for sale and described it to me in detail so I was sure we were both talking about THE Master Edition.  I asked him where it would be when the store opened so I knew exactly where to find it.  He was very helpful, but there was too much at stake to leave this up to chance, so I decided to visit the store that evening before they closed.  I put on 2-5 layers of clothing, packed my sleeping bag, some snacks, some drinks, and an ebook reader and headed out.  When I arrived at the store, I couldn't find the gentleman I had spoken with on the phone, but was able to get the manager to confirm for me that they had at least one Master Edition - she also described it in detail and followed up with a very thorough explanation that it couldn't be held, reserved or otherwise spoken for.  They would be passing out tickets for the consoles themselves in the morning but not for anything else i.e. not for the Master Edition.  But IT WAS THERE!!!! Originally I had planned to maybe hit the midnight opening at the other Best Buy and head to this one after, but the possibility of someone beating me to that Master Edition made me resolve then and there to plant myself in front of the door and wait.  I was just going to be spending the entire interval obsessing over it anyway - might as well do something useful with that energy. That was just before 8PM.  The store would not open again until 10AM.

     So I rushed to the car, pulled out my supplies and set up right in front of the entrance for a 14-hour wait to save both the integrity of my Zelda collection, and my 21-year streak of having every new Nintendo platform on day 1.

     I'm no stranger to waiting in line for stuff like this.  I did it for the PS2, the DS, the PSP, the Wii, and the PS3.  On every one of those occasions there were always people driving by staring.  Many would roll the window down and ask what we were waiting for, and in almost every instance, someone would make fun of us for waiting in line or call us crazy.  This time my experience was a lot different.  I had set up about an hour before the store closed for the night so around a dozen customers came and went in that time.  The ones that bothered to acknowledge my presence would pause, cock their head and then say something like "Oh, he's going to get a Switch." And to my surprise it was almost always followed by something complimentary like "That's dedication." or "He's gonna get one."  As the Best Buy employees left, they would sometimes stop to make sure I knew that the store wasn't opening at midnight, or to tell me they were impressed with my determination.  The manager I had spoken with earlier left at around 10:30PM and took the time to tell me cheerfully that she'd see me in the morning, and asked if I was going to be warm enough.  I was impressed by the level of awareness and acceptance of everyone who stopped by.  Waiting in line for over a dozen hours for a toy was no longer a fringe activity, it was accepted and even lauded.  Some things really do change.

     There were a couple of people that stopped by around 11 to get in line with me but they were completely bewildered when I told them that this particular Best Buy was not opening at Midnight and left shortly afterward.  It wasn't until around midnight that the next guy crazy enough to wait all night showed up.  He was a night-shifter so he didn't get off of work until 11PM and knew that was way too late to get in line for a midnight opening.  He was woefully unprepared for how cold it was outside.  Truthfully I was still freezing underneath all of those layers.  20 degrees to 98.6 is one heck of a differential.  We chatted for a hour or so.  I offered to hold his spot in case anyone else showed up so he could use the restroom at a nearby gas station.  He returned about 30 minutes later and handed me a packet of chemical hand warmers (man was that a kingly gift).

     My wife would text me occasionally - she had graciously offered to stop by and hold my spot while I used the restroom and/or warmed up in the car.  Around 3AM I took her up on it.  After a brief bathroom break I was right back at it.  Shortly after that the other guy in line left to go somewhere and sleep, leaving the seat from his van next to me as a placeholder.  He came back around 6AM and I was still the only person in line.  Around 7 my wife came back to give me another bathroom break.  It wasn't until around 8 that others began to gather around and wait in line.  My wife came and carried away the chair and sleeping bag around 9AM so I could stand for the last hour of waiting - and the last hour is ALWAYS the longest...

Safely Nestled in My Car - the Spoils of Victory
     As promised, a manager came out at 9:30AM to hand out tickets for consoles.  People wandered back to their cars for warmth, but, of course, I wouldn't budge from the front of the line.  When the doors finally opened at 10, I rushed inside to the spot where the manager had told me it would be, and there it was - the Master Edition, the one and only Master edition in all it's glory.  The manager was standing beside it ready to guide me to it on the off chance that my eyes hadn't been drawn to it like rare earth magnets to a refrigerator.  I had been unable to talk them out of putting one of those terrible anti-theft devices on it so there were little indents along some of the edges, but after what I went through, those are battle scars!  I snatched it up, and heard to the disappointed voices of the people behind me asking if there were any others.  I rushed it over to the counter and begged the clerk to immediately remove the anti-theft device before it could do any more damage.  I didn't even bother to pick up my console until the Master Edition was safely in my car.  After tucking it gently into my trunk, I headed back inside to redeem my coupon for a Switch. While inside, one of the other managers spotted me and since I wasn't carrying it anymore asked almost in concern approaching panic if I had gotten my Master Edition. She didn't realize I had left and come back already. Don't think it didn't enter into my thoughts that someone might have been desperate enough to break into my car to get that Master Edition!

     It was done - launch day had been saved! I ended up with the Master Edition and a console on day 1. Thank the maker.

     The epilogue to my story was that on the way back home, I swung by Gamestop to pick up my Special Edition of Zelda (the less-mega-rare premium version) and subsequently spent the weekend absorbed in the new Zelda game and really enjoying the new console.

     Amazon had botched the Switch launch so badly that I now count it as a blessing that I didn't manage to get a pre-order in with them.  They actually ended up losing my Super Bomberman R order altogether.  When Monday came, my pre-ordered console from Target arrived and I promptly sold it (at cost) to a friend who wasn't able to get one.

I love happy endings:)


     

Monday, February 06, 2017

Unmounting a volume with open file handles to deleted files without killing the process

Recently we've been turning on compression in our Oracle databases because it saves us a ton of disk space and actually improves the performance of the databases.  Among other things the process involves writing the newly compressed tablespaces to new data files (the systems in question are not using ASM).  Because we're moving the DB's from where they are, that gives us the opportunity to do some housecleaning before moving them back, such as un-mounting, checking and possibly resizing the existing filesystems.

Occasionally it will occur that after the data has been migrated away and the old data files deleted, that Oracle still has open handles to the deleted files.

So the DBA tells me the filesystem is clear and I can have it, but then this happens:

[root@kwt-r3oql00 E1Q]# umount /dev/mapper/vg_kwt_r3oql20_s00-oracle_E1Q_sapdata7
umount: /oracle/E1Q/sapdata7: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))

And when we go to see who the culprit is:

[root@kwt-r3oql00 E1Q]# lsof /oracle/E1Q/sapdata7
COMMAND     PID   USER   FD   TYPE DEVICE    SIZE/OFF     NODE NAME
oracle_13 13214 oracle  520u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_13 13254 oracle  519u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_13 13258 oracle  520u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_13 13286 oracle  517u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_13 13294 oracle  520u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_13 13526 oracle  506u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_13 13996 oracle  512u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_14 14424 oracle  493u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_14 14488 oracle  515u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_14 14574 oracle  403u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_14 14712 oracle  506u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_23 23296 oracle  273u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)
oracle_24 24260 oracle  277u   REG 253,17 10485768192 17317890 /oracle/E1Q/sapdata7/sr3usr_1/sr3usr.data1 (deleted)


This is super annoying because the DB seems to have forgotten about those open handles and internally has no mechanism for gracefully releasing them.  If you can tolerate the outage restarting the DB will release those handles, but if you can't there is another solution.

The GNU debugger (gdb) will allow you to attach to the process and close the file handles.  It's not without risk, but I've been using it successfully for years.  The main drawback is how tedious it is to sift through /proc looking for the open file handles and closing them one by one with the debugger interactively.

Laziness being the father of invention, I wrote the following script to do this for you.  Simply provide it the path to check for open deleted file handles, and it will find the processes, attach the debugger and remove the handles.


#!/bin/bash

FS=$1

if [[ -z $FS ]]; then
   echo "Please provide a filesystem path to check for open file descriptors to deleted files"
   exit 1
fi

echo "WARNING: This is super-dangerous.  Please don't use it in Prod without a "
echo "         really good reason and a change request/blackout"
read -p "Type 'C' and Enter to continue, anything else to abort: " CHECK

if [[ "$CHECK" != "C" ]]; then
   echo "Aborted by user.  No changes made."
   exit 0
fi

# Get a list of processes with open file handles to the given directory
PIDLIST=`lsof $FS | egrep -v 'PID' | awk '{print $2}' | sort -u`


for PID in $PIDLIST; do

  unset DESCLIST

  # Get a list of file descriptors for that PID that refer to deleted files
  DESCLIST=`ls -l /proc/${PID}/fd | grep deleted | awk '{print $9}'`

  # Display the list
  echo "The Process $PID has open file descriptors for deleted files:"
  echo "${DESCLIST}" | sed 's/^/  /g'

  # Create a name for the script for gdb
  DEBUGSCRIPT=/tmp/.$PID

  # Remove any previous version of the script
  if [[ -f $DEBUGSCRIPT ]]; then /bin/rm $DEBUGSCRIPT; fi

  # Write a close command for each deleted file descriptor
  for DFD in $DESCLIST; do
     echo "p close(${DFD})" >> $DEBUGSCRIPT
  done

  # Detach and close the debugger
  echo "detach" >> $DEBUGSCRIPT
  echo "quit" >> $DEBUGSCRIPT

  echo "Forcibly closing handles for deleted files on process $PID"

  # Run the debugger in batch mode to execute the script and close the handles
  /usr/bin/gdb --pid $PID --batch -x $DEBUGSCRIPT

  # Wait before deleting the script
  sleep 1

  # Clean up the script
  /bin/rm $DEBUGSCRIPT

done



Tuesday, January 31, 2017

New Nintendo 3DS XL motherboard swap

The Super Famicom is a contender for my favorite game console of all time.  I loved the North American redesign as the Super NES, but the Super Famicom was what I fixated on for years leading up to its release.  Everything from the contours of the console to the colors of the buttons on the controller seemed so lovingly and carefully designed.  Nothing before or since has really come close.

So when Nintendo announced in early 2016 that there would be a limited edition New Nintendo 3DS LL (that's what the XL is called in Japan) to celebrate the anniversary of the Super Famicom, I immediately pre-ordered it from National Console Support, then dutifully waited 8 months until it arrived.  Whenever a new 3DS special edition is announced in Japan, there's usually a bit of uncertainty about whether it will also come out in the U.S., but this one was clearly not going to be brought here.  It's doubtful most Americans would even recognize what it was supposed to be.

When it arrived, I was thrilled.  Even the product box was a callback to the original Super Famicom box, with the simple stylized stencil design and gray background. 

But enough about that, the N3DS Super Famicom edition is awesome, etc.

Problem is that Nintendo, in a colossally frustrating and reviled move decided that the 3DS would be their first handheld ever to be region locked, so basically I had this cool handheld that would only play Japanese games.  I don't actually own any Japanese 3DS games, and what I really want is to play my US games on this system.

In olden times when we wanted to get around a region lock, we bought and installed mod chips or used boot disks, and more recently used softmods.  Unfortunately Nintendo did a really thorough job of embedding the region lock.  There are, apparently, a couple of region bypass methods for the 3DS, but they all require that you be on some old version of firmware and never update your system, and/or that you have a copy of some obscure game like Cubic Ninja which is only expensive because people want it to help trigger an exploit.  These hacks are involved, not permanent, and always in danger of getting patched out in a future firmware release.  What's most ridiculous about all this is the fact that even flash carts which allow you to play illegally copied games all day long are STILL stymied by region locks.

So the next best thing is to put a U.S. brain (a.k.a. motherboard) into a Japanese 3DS.  When the New 3DS XL was brand new this was a pretty expensive proposition as the only way to get such a motherboard was to buy a domestic New 3DS XL.  However after a year or two a lot of these have been damaged to various degrees and you can get a broken one much more cheaply.  In my case a repair shop was parting out broken units and I was able to get a working motherboard for just over the price of a new game.

The process of transplantation is straightforward but a little tedious. There are plenty of guides online about teardown, and even some videos showing disassembly and reassembly but they're all very terse and leave out little nuances.

There are over a dozen delicate little connectors that need to be carefully removed before the old motherboard will come out and almost all of them are attached to flimsy little ribbon cables.  Every instruction I've ever read that involved removing or replacing ribbon cables came with a stern warning about how easy they were to damage and basically made it seem like they were made of tissue paper. I've done a lot of repairs where I had to twist or pull on the ribbon cables to get them into position, and in a couple of instances I've accidentally pulled too hard or twisted one badly enough I was sure it wouldn't work when everything was back together, but in every case they were fine. I've never actually damaged one. Maybe that's because all the warnings made me extra careful?

Anyway, there are several different types of connectors and they each have to be removed a little differently.

The L and R button clusters are connected to the back of the unit with little sockets. You can pry them up easily with a fingernail or small flathead screwdriver.  The cart slot is also connected to the motherboard with a larger version of the socket connector.

The WiFi antenna (the red wire) is a connected by a standard socket - the same kind you'd see in a laptop.  It's pretty sturdy, just pry it off.


There are four kinds of ribbon connectors - I've color coded them in the image above.

The narrow all-black connectors (outlined in red) just use tension - gently pull them straight out and gently push them straight in. For the narrow black connectors with a large hinged gray lever (outlined in gray) gently pry up the gray lever from the opposite side of where the cable is inserted and the cable should just slip out. (The analog stick ribbon is folded over one of these - you'll need to unscrew the backing for the analog stick and pry it up before you'll be able to get at its ribbon connector.

The wider levers are color-coded according to which direction the connectors should be inserted. The dark sockets with light colored levers (outlined in blue) indicate the ribbon needs to be inserted contact (i.e. shiny) side up.  The light colored sockets with the dark levers (outlined in green) indicate the ribbon needs to be inserted contact side down. There are two wide levers not shown in the picture that are underneath the board in the upper-right corner. One of the two on the back side will actually fit either way - be careful and follow the color code.

The trickiest part of getting the thing apart and back together involves the 3 ribbon cables going into the hinge in the upper right (as your face the back). These ribbon cables are literally right on top of each other and have very little slack to work with.  What's worse is that they are constantly in the way of each other when you try to reassemble them.  (I had to resort to using a sliver of tape to hold one of them out of my way while I worked on the first ribbon.) If you don't have small fingers (and I very much don't) you'll definitely want a good precise pair of angled tweezers. The ribbon on the very bottom (on the underside, the longest of the three that connects perpendicular to the others) is jammed into the hinge recess at an odd angle so when it comes undone, it's not intuitive how it's supposed to go back in.  This is where the color coding for the connectors comes in really handy - it has a black hinge on the socket, so it's conductor side down (in relation to the surface of the board).  I strongly recommend taking pictures before removing the ribbons so you have a reference for how to fold them back in.

If it's not obvious from the subtext, I managed to insert one of the ribbons backwards and the result was that when I turned the system on, the power light would come on and after a second or two there would be a low pop and the system would power down.  Fortunately this didn't cause any apparent damage.  When I fixed the cable orientation everything came up normally and has been working perfectly for a couple of weeks with no issues.








Online Resizing of a Boot Partition on RHEL 6 or 7 with parted.

The Problem:

I've been getting away with using a 200MB boot partition for over a decade, but RHEL 7 has started pushing the limits of that scheme.  Between GRUB2 and the automatic creation of a rescue kernel and initramfs that's twice the size as a normal initramfs, that 200MB gets consumed much more quickly.  I can only make it through about two update cycles before having to uninstall the old kernels to avoid upgrade failures.

This is a handy command for removing all of the old kernels by the way:

yum erase $(rpm -qa kernel* | egrep -v $(uname -r))


I've updated my provisioning tools to make the boot partition larger, but obviously I still have to do something about the pre-existing systems.


All of my VMs are built with provisioning templates that create a boot partition on the first disk, then format the remainder of the first disk for LVM.  In order to expand the boot partition, it's boundary needs to be moved into space occupied by the OS itself.  Fortunately LVM is pretty flexible.  Not only can I "move" the partition boundary, I can do it while the system is up and running.

Here's what we're starting off with:

[root@azrst2ill001 ~]# parted -s /dev/sda print
Model: Msft Virtual Disk (scsi)
Disk /dev/sda: 53.7GB
Sector size (logical/physical): 512B/4096B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  211MB   210MB   primary  xfs          boot
 2      211MB   53.7GB  53.5GB  primary               lvm


Two partitions on the first disk, the first partition is for /boot, the second is owned by LVM. As you can see there's no space between them: the second partition begins exactly where the first ends.


Here's what LVM looks like:

[root@azrst2ill001 ~]# vgs
  VG        #PV #LV #SN Attr   VSize   VFree
  rootvg      1   6   0 wz--n-  49.80g 21.94g

[root@azrst2ill001 ~]# pvs
  PV         VG        Fmt  Attr PSize   PFree
  /dev/sda2  rootvg    lvm2 a--   49.80g 21.94g


The primary volume group with all of the OS volumes is named "rootvg" and it is contained on a single PV - /dev/sda2.  With some quick mental math you can see it's consuming just over 30GB.

Note:These instructions assume the machine was built with the default filesystem for each release, so the RHEL 6 instructions assume you built /boot with ext4, otherwise the instructions assume you're working with RHEL 7 and built /boot with xfs.

Warning:
This process is somewhat dangerous because if it is done incorrectly or the server is rebooted before it is complete, the system could be unbootable..


The Goal:
Expand the boot partition from 200MB to 1GB without re-imaging the system.


The Solution:


1. Verify the volume group has at least 1GB free because that's basically where the extra space for /boot is going to come from.

vgs


2. Add a temporary disk to the machine that's at least as large as the root volume group.  In my case I attached a new virtual data disk in Azure and 128GB was the smallest option.

UDEV created the device as /dev/sdc:

[root@azrst2ill001 boot]# lsblk
NAME                          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                             8:0    0   50G  0 disk
├─sda1                          8:1    0  953M  0 part /boot
└─sda2                          8:2    0 49.1G  0 part
  ├─rootvg-rootlv (dm-0)      253:0    0    8G  0 lvm  /
  ├─rootvg-localhomelv (dm-1) 253:1    0    2G  0 lvm  /usr/local/home
  ├─rootvg-optlv (dm-2)       253:2    0    5G  0 lvm  /opt
  ├─rootvg-swaplv (dm-3)      253:3    0    2G  0 lvm
  ├─rootvg-tmplv (dm-4)       253:4    0    5G  0 lvm  /tmp
  └─rootvg-varlv (dm-5)       253:5    0    4G  0 lvm  /var
sdc                             8:32   0  128G  0 disk
sdb                             8:16   0   56G  0 disk
└─sdb1                          8:17   0   56G  0 part /mnt/resource




3. Add the new disk to the primary VG

vgextend rootvg /dev/sdc


4. Move all of the physical extents (the data basically) off of /dev/sda2. This process will take some time.

pvmove /dev/sda2 


5. Once all of the data is cleared off of /dev/sda2, delete it from the primary VG.

vgreduce rootvg /dev/sda2


6. Just for good measure remove /dev/sda2 from LVM altogether

pvremove /dev/sda2


7. (Optional) At this point I wanted to make sure I had some kind of fallback if I dorked something up, so I made a couple of quick backups.

Firstly, I backed up the whole boot partition:

dd if=/dev/sda1 of=/root/bootpart.img

Next I backed up the MBR

dd if=/dev/sda of=/root/mbr.img bs=512 count=1

These backups are only really helpful if you realize you screwed something up before trying to reboot.


8. Delete the LVM partition

parted -s /dev/sda rm 2


9. Put the starting position of the boot partition into a variable

START1=$(parted -s /dev/sda print | grep "^ 1" | awk '{print $2}')


10. (The scary part) remove the boot partition.  

You can get away with this because the only thing parted is manipulating is the partition table.  The actual filesystem data is unaffected by this.  As long as you put the start of the boot partition in exactly the same place (which is why we saved that place in a variable before deleting it) everything will line up and work normally.

umount /boot
parted -s /dev/sda rm 1


11. Re-create the boot partition

parted -s /dev/sda mkpart primary xfs $START1 1GB

If you're doing this on RHEL 6, you probably want to go with ext4 instead:

parted -s /dev/sda mkpart primary ext4 $START1 1GB


12. Set the partition to bootable

parted -s /dev/sda set 1 boot on


13. Check the filesystem.

xfs_repair /dev/sda1

Note: if you get an error message  "/dev/sda1 contains a mounted filesystem" just run "umount /boot" again right before you run the command. For some reason RHEL 7 kept re-mounting /boot automatically on me(autofs isn't even installed).  I didn't take the time to figure out what was causing this behavior, but it's easy enough to work around.

If you're doing this on RHEL 6:

fsck -f /dev/sda1


14. Mount the /boot filesystem (if it didn't already mount itself)

mount /boot


15. Resize the /boot filesystem to fill the new larger partition.

xfs_growfs /dev/sda1

If you're doing this on RHEL 6:

resize2fs /dev/sda1


16. I want to use the rest of the space on /dev/sda for the second partition.  To get the start and end positions of free space, I used parted to set some variables.

START2=$(parted -s /dev/sda print free | grep Free | tail -1 | awk '{print $1}')
END2=$(parted -s /dev/sda print free | grep Free | tail -1 | awk '{print $2}')


17. Create the new LVM partition (the -- tells parted to accept a blank value for filesystem because we're going to use the partition for LVM instead of installing a filesystem onto it.)

parted -s /dev/sda mkpart primary -- $START2 $END2


18. Set the partition flag to LVM

parted -s /dev/sda set 2 lvm on


19. Check to make sure everything looks right - most importantly that the boot partition is showing the desired new size, and that it's set to bootable, and that when you do an ls on /boot you can see the kernel and intramfs files.


[root@azrst2ill001 ~]# parted -s /dev/sda print
Model: Msft Virtual Disk (scsi)
Disk /dev/sda: 53.7GB
Sector size (logical/physical): 512B/4096B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  1075MB  1074MB  primary  xfs          boot
 2      1075MB  53.7GB  52.6GB  primary               lvm



[root@azrst2ill001 ~]# ls /boot
config-3.10.0-327.10.1.el7.x86_64                        initrd-plymouth.img
config-3.10.0-327.28.3.el7.x86_64                        symvers-3.10.0-327.10.1.el7.x86_64.gz
grub2                                                    symvers-3.10.0-327.28.3.el7.x86_64.gz
initramfs-0-rescue-edcc5bfcc87f4b90a0ae36219b0138e3.img  System.map-3.10.0-327.10.1.el7.x86_64
initramfs-3.10.0-327.10.1.el7.x86_64.img                 System.map-3.10.0-327.28.3.el7.x86_64
initramfs-3.10.0-327.10.1.el7.x86_64kdump.img            vmlinuz-0-rescue-edcc5bfcc87f4b90a0ae36219b0138e3
initramfs-3.10.0-327.28.3.el7.x86_64.img                 vmlinuz-3.10.0-327.10.1.el7.x86_64
initramfs-3.10.0-327.28.3.el7.x86_64kdump.img            vmlinuz-3.10.0-327.28.3.el7.x86_64



20. Move the rootvg volume group back to the first disk, and remove the temporary disk from the VG.

vgextend rootvg /dev/sda2
pvmove /dev/sdc
vgreduce rootvg /dev/sdc
pvremove /dev/sdc


At this point it should be safe to remove the temporary disk you added in step 2, and to delete the backup .img files if you chose to create them in step 7.


Even though you can perform this whole operation with no downtime to the system, I strongly recommend rebooting at this point to verify everything is working as it should.  Better to find out now than months from now when it's not fresh in your mind.

Wednesday, November 30, 2016

DVL-909 Repair Notes

The unit was purchased on Ebay for about $10.  The seller basically said he pulled it out of storage and it just didn't work.  Images in the listing showed both the LD and DVD trays ejected.


When it arrived, as per usual, the packing job was not adequate to protect the unit from damage.

The machine powered up but the tray would not eject.

Disassembled the unit and found the tray was jammed by the yellow switch lever that switches the eject mode from LD to DVD.  It had evidently received a shock hard enough to push it up into the workings of the tray itself.  Carefully prying it loose allowed the tray to be manually ejected by finger-turning the pulley just under the left side of the tray at the front of the unit.

The next issue I noticed was that the pickup had received a shock hard enough to move the left side up above its track so that the entire LD pickup was sitting at about a 30 degree angle.  Miraculously the motor holder was not damaged at all.  I removed the LD tray, then  removed the bottom rail screw to loosen the rail and free the LD pickup, replaced the pickup onto its track correctly then re-fastened the rail screw.

After replacing the parts and restoring power, the player was able to properly eject both trays with button presses, and the pickup moved freely along the track.  The player also correctly switched between the LD and DVD pickups.

The next issue - and probably the one noticed by the seller, was that the spindle motor was not engaging.  A simple continuity test of the pickup motor's socket (green and white wire going to the power supply) revealed that the motor was not making contact internally.

To remove the spindle motor, it was necessary to disassemble the top of the player and remove the entire pickup assembly.  The procedure was to remove the ribbon cables and sockets from the board to the right of the H-bar, remove the H-bar, remove the narrower of the two ribbon cables toward the front of the player (the wider one simply routes under the pickup assembly to the power supply - not necessary to remove it), the spindle motor socket, and the two ribbon cables toward the back of the assembly.  The "box" toward the rear of the assembly was removed, and the springs holding the assembly up were unfastened from their holders.  The pickup assembly is held in by six screws - these screws are slightly longer than the other screws and should be kept separate. The two screws toward the front are recessed very deeply and require a longer philips head screwdriver than typical to remove.

Once the assembly was removed I removed the three screws securing the spindle motor, removed the power wires from the routing clips and very carefully worked the spindle motor out of the assembly.

Once removed, I used a metal cutting wheel and dremel to cut a slot in the motor, and grind down the tension clips holding the bottom of the motor in place.  Once pried open I noted that the motor brushes were completely corroded.  After a vain attempt to repair the brushes with solder, the motor only worked for a short time before failing again.  Round about that point I noticed that Pioneer evidently used the same spindle motor in quite a few different models of LD player.  I was able to take the spindle motor from a different player with an electrical failure (a CLD-D605) and install it in the DVL-909.

With the replacement spindle motor installed and the player re-assembled, it was able to play Laserdiscs again.  Inserting a DVD caused the unit to switch to the DVD pickup, however the pickup was not functioning correctly, so it would simply report "NO DISC".  LD playback seemed normal except for side B playback where it would "skip" during the first couple minutes of playback.  Fast forwarding past the first couple of minutes resulted in normal playback for the rest of the disc.

My research into the skipping problem led me down several false paths.  I checked the upper gear track for debris and cleaned it with a toothbrush.  I cleaned and re-greased the upper rails and track.  When none of these things helped, it was suggested the M-holder (the plastic piece that holds the motor and the plastic gears that move the pickup) was damaged.  This seemed plausible because the pickup had been forced out of alignment violently during shipment - although I could see nothing wrong with it, it was possible it was cracked somehow.  I found a place called "Donberg" in Ireland that still stocked the M-Holder and the gears that go with it, and ordered replacements.

When the new parts arrived, I removed the LD pickup, thoroughly cleaned it, including removing the ribbon cable and cleaning it with Deoxit.  After installing the new M-holder and gears, the skipping during the first couple minutes of side-B persisted.  I removed the plastic upper rail and thoroughly cleaned it.  I readjusted the centering screw on the upper part of the pickup, but still the issue persisted.  The plastic rail is attached with clips and it hangs about half a millimeter from the top of the pickup assembly when installed.  Manipulating this by pulling the plastic gear guide up during playback seemed to allow the player to get past the skipping, but propping it up with twist ties did not allow it to play normally.

Eventually I discovered the problem was the little lever that acts as a brake and triggers the limit switch on the upper pickup assembly.  The brake was wedging itself against the pickup creating just enough tension to prevent it from moving normally during playback.  After trying to lubricate it with no success, I carefully removed the lever and examined the end which makes contact with the pickup.  There was a clear scuff mark where it had been wedging against the pickup.  I very carefully filed it down - barely enough to level off the top, then used a dremel polishing wheel and some polishing compound to smooth it completely.

This is looking up at the upper track.  The brake lever, circled in red, moves forward (to the right as this image is oriented) when the pickup reaches the limit of travel. The brake is supposed to both stop the pickup from moving and depress the limit switch. In this case the surface of the brake that came into contact with the pickup was sticking, preventing the pickup from moving back (to the left as this image is oriented) smoothly and causing the image to video to "skip" back randomly during the first couple of minutes of side B playback.

After re-assembly the system developed a new fault - the LD pickup became stuck in the rotator at the rear of the pickup assembly.  It would bounce back and forth furiously for a few seconds and give up.  I disassembled the upper part of the unit to get room to work then manually turned the worm gear on the pickup motor to get it back onto the track.  It gave quite a bit of resistance like some debris had become stuck in the gear track.  Although I never did see anything, I cleaned the section where it had become stuck thoroughly with a toothbrush and that seemed to clear the problem up.

Once the LD pickup stopped getting stuck in the rotator, I was able to confirm that the side-B skipping problem had also been resolved.

I noticed that the DVD pickup moved somewhat erratically and would occasionally get stuck when positioned all the way forward.  Getting the pickup out was a little tricky because the player does not like to eject the LD tray while the DVD pickup is on the bottom.  Getting it out required that I place a DVD in the player and wait for it to move the DVD pickup forward, remove the power cord, remove the crossbar and spindle clamp, then remove the DVD by hand, plug the player back in, then press the LD eject button and quickly remove power before the pickup makes it all the way back into the rotator, then manually eject the tray. This could probably have been done more easily via the service menu, but I did not have a means of using the menu at that point.

I removed the DVD pickup, then removed and cleaned its M-Holder and gears.  Next I used Deoxit to clean the ribbon cable and socket.  I used a q-tip with alcohol to clean the disc sensors and gently swab the dust off of the lens.  Once re-assembled the DVD playback functioned normally.

Unfortunately the front panel of the player offers no way of pressing "Enter" on a DVD, and the player came without a remote control.  Fortunately, however, Pioneer likes to re-use remote codes, and I was able to use the remote from my DV-525 DVD player to not only interact with DVD menus, but also access the on-screen player settings.

Thursday, November 10, 2016

For the Love of Laserdisc

Why Laserdisc?

Plenty of obsolete formats garner a following because of nostalgia (I can't fathom why else anyone would actually like 8-track cassettes), and a few of them remain popular long after the sun sets on their era because they remain superior in one way or another to the thing that replaced them.

Laserdiscs kind of fit in both categories.  I didn't grow up with laserdiscs.  Heck, I only ever remember seeing a laserdisc player once when I was growing up.  But I sure knew what it was when I saw it.

Dipping My Toe

In the late 90's when DVD was on its meteoric rise as the replacement for both Laserdisc and the vastly more popular VHS, I dabbled a little with laserdiscs.  Being a 12 inch disc format just like vinyl records, the disc jackets featured a whole square foot of artwork.  They were just cool to look at.  So, like a lot of people, I bought a handful of laserdiscs before I even had a player to play them with.

Some time in the late 90's, My wife and I stopped into a rental store that was liquidating its laserdisc stock.  Most of it had been picked over by the time we arrived, but I managed to pick up the whole Ninja Turtles trilogy for about $15.  Sometime later I was browsing the case at a Goodwill and found a sealed copy of the T2 special edition (it was exclusive to LD at the time).

A buddy from work was a real AV enthusiast and was constantly educating me on the virtues of Laserdisc, and DVD.  With Laserdisc being replaced by DVD, people were selling their players like crazy so I picked out a player on ebay and eagerly waited for it to arrive.  My buddy lent me a couple of discs to break in the new player.  One of them was Con Air - I can't recall the other.

Well, the player arrived, I got it all set up, watched one movie, started a second and the player broke.  After a heated discussion with the seller I sent the player back and decided not to jump into the format after all.

Six months later I had switched jobs and was recounting the experience to a co-worker who offered to sell me his barely-used Laserdisc player for $100.  I took him up on it and I've had a great working player ever since.

Even so, I didn't really do a lot of collecting.  I picked up a couple odds and ends.  When the Transformers: The Movie was re-released by Pioneer Japan, I somehow got wind of it and ordered a copy which is still sealed to this day.  When browsing the LD collection at a local used record store I found a copy of Anne of Avonlea for my wife.  You get the idea.

Another Angle

About four years ago, I decided that I really wanted to try out the Pioneer LaserActive system - basically they made a Laserdisc player with an expansion port for video game hardware modules.  There was a Sega Genesis/Sega CD version and a TurboGrafx/TGCD version.  In addition to being able to play cartridge and CD games on the player, it was also possible to play special games on Laserdisc.  Because Laserdisc is an analog format, these games haven't really been emulated, and can't really be copied so unlike just about every other video game on the planet, these games can really only be played on real hardware from retail discs.  As you might imagine, these games and players are pretty rare and expensive.  I only own one game.

Renewed Interest

Earlier this summer a friend from work asked me if I knew where his buddy might buy a laserdisc player.  The buddy had just bought a copy of the Star Wars Definitive Edition on Laserdisc and didn't have a player to play it on.  I had been looking for an excuse to come over to his house and see his theater setup (140-inch screen was all I needed to know), so I offered to bring one of my players over so we could all watch some un-"fixed" Star Wars.

The experience turned out to be immensely fun and educational.  Firstly the deluxe box set of the Star Wars trilogy was just really really cool.  Secondly the ability of modern A/V equipment to clean up and scale composite video from a Laserdisc to 1080p is surprisingly good.

The theater was equipped with a contemporary Onkyo receiver that was 1 model different from the one I use.  I had never bothered to connect a LD player to mine because I assumed the picture would look like crap no matter what I did.  I incorrectly assumed this because my primary hobby is video games and connecting old video games to modern AV equipment usually does result in extremely poor picture quality.  I have to use a lot of specialized AV equipment to get a good picture out of old game systems on new TVs.  In the back of my mind I knew this was because old game consoles took advantage of characteristics of CRT televisions that weren't necessarily part of the video specification of the time.  When non-CRT televisions became the standard these tricks no longer worked because the scalers built in to modern TVs and AV receivers were not designed to process the out-of-spec signal.  What never connected with that knowledge until I was sitting in my friend's theater watching Star Wars on his 140-inch screen and wondering how it could look so good, was that this is EXACTLY what those scalers are designed to handle.  And boy do they do it well.

The event also opened up discussions where I was able to share my knowledge of the LD format - how it's recorded in composite and not RGB, how the CAV and CLV formats work etc... This got me onto the topic of AC-3 (also known as Dolby Digital) and DTS audio and basically where the Laserdisc technology peaked before its obsolescence.  Having these facts rattling around again reminded me that I had always regretted in the tiniest way never having been able to experience everything LD had to offer.  So, of course, 2016 was the perfect year to rectify that.

Dolby Digital AC-3 and DTS

The LD format had a long and storied history and evolved quite a bit along the way.  The fact that the designers were able to take what started as a fully analog format and add digital audio, and then later multi-channel digital audio (Dolby Digital and DTS both supported up to 6 discrete channels of audio) spoke volumes about the versatility of the technology.  What was originally only able to handle less than 30 minutes per side of grainy muddy video was stretched and stretched and refined until ultimately each side could contain up to an hour of high-quality video.  But this evolution was not without its price.  The very first players couldn't play digital audio.  While most players manufactured from the early 80's onward do support digital audio, only a handful of them support AC-3 and DTS, and there's a huge catch even with the ones that do.

AC-3 is recorded on the disc on one of the analog audio tracks as a matrixed radio frequency signal.  It sounds like static when played as analog audio.  In order to actually get discrete 6-channel audio (so-called "5.1" because it carries 5 full frequency channels and a subwoofer channel with just the low-end frequencies) from AC-3 one was required employ a number of devices at once.  Firstly the player had to recognize the AC-3 signal and output it as radio-frequency (RF) instead of analog sound.  Secondly the RF signal needed to be demodulated - that is taken from matrixed RF to a digital bitstream.  Because it would make the players more costly and the number of discs to actually use AC-3 was relatively low, the circuitry needed to demodulate the AC-3 RF signal was never built into the players.  Instead users either had to buy an external demodulator (which weren't cheap) or buy an A/V receiver with the circuitry built in (also not cheap because it was a feature reserved for high-end receivers).  And finally, the digital bitstream had to be decoded - that is turned back into multiple analog audio channels to be output from your surround speakers.  Fortunately that last step is something that pretty much every digital A/V receiver ever released is capable of doing because the digital bitstream output by the decoder is the same format used by DVDs with a DD soundtrack.

To make things even stranger, competing with AC-3 was DTS.  DTS was also a 5.1 channel format but a decidedly simpler one.  Instead of using an RF-encoded analog audio track, DTS used both of the digital stereo tracks to carry a bitstream which could then be decoded by pretty much any A/V receiver.  This had the dual advantages of much higher bandwidth than AC-3 and the player didn't need any proprietary circuitry or really even be aware of DTS to support it - it just had to provide a digital audio output.  Higher bandwidth is a big deal with audio formats because less compression is needed, and less compression means wider dynamic range and higher fidelity.  Because of this DTS laserdiscs are among the most sought-after and most expensive ones to find used.

In order to support both DTS and AC-3, you would have to use two distinct digital outputs from a compatible LD player - an AC-3 RF output, and a digital audio output.

Plunging In, Head First

After doing my research and shopping around, I settled on a plan to start enjoying the full digital surround sound experience on Laserdisc.

First I followed this extremely helpful list of AC-3 RF Audio Gear: http://www.avsforum.com/forum/90-receivers-amps-processors/652457-list-ac-3-rf-audio-gear.html

The most cost-efficient (but least space-efficient) way to demodulate the RF signal was to buy an old AV receiver which not only had the demoduator built in, but also featured a digital audio output.  Many of the higher-end models featured audio pre-outs so the receiver could be used as a preamplifier for external high-end amplifiers.

I found such an AV receiver on Ebay for a reasonable price and pulled the trigger.

Next I needed to get an LD player that supported AC-3 AND had a digital audio output.  There may be a handy list out there for that combination, but if there is I couldn't find it.  Instead I browsed through tons of Ebay listings and built a list of eligible models based on photos of the backs of the players.

I settled on a model, found a good deal and bought it.

My collection only had a single movie encoded with AC-3 (Braveheart), and none with DTS.  The cheapest thing I could find with a DTS soundtrack was Con Air!  How cool was that?

Lastly I needed to work out how to cable everything together.  I dug through my boxes of old cable and scraped together just enough to connect everything up and test with (that is to say, on the floor in front of my other receiver).  Then I headed over to Monoprice to order new cable in the required lengths so I could install it permanently.

About a week later, everything arrived on the same day.

Fixing Your Player Is Part of the Fun

Unsurprisingly the LD player was non-functional when it arrived.  If you ever get a wild hair to get into this hobby, this is just something you need to get used to. These players are old, and while they may be sturdy, they do not ship well.  The shipper had done an amazing job packing it, but the loading mechanism was stuck - probably because of a decade of sitting idle on a shelf somewhere.  I dismantled the player and started down a rabbit hole that was deceptively deep.  After studying the mechanism and getting a rough idea how it was supposed to work I managed to get it free, but this only led to the next failure. I'll spare the gory details and just say that I was able to get the player functioning again after about 30 minutes of tinkering.

By that evening I had experienced my first AC-3 laserdisc, and it was fantastic.  The Dolby Digital track on Braveheart sounded better than anything I had heard since seeing the film in the theater.  When I looked into why this might be, I discovered that, in practice, the AC-3 tracks on LD are generally far less compressed than they are on DVD.  The difference is amazing.  As to why it sounded better than even the Blu-ray release, I have to chalk that up to mixing.  My theory is that since AC-3 was a new and high-end feature during the heyday of Laserdisc, the audio was mastered with a slant towards showing off - adding a little extra reverb here or there or putting your subwoofer through its paces to remind you you're listening to "surround" sound.  Whereas with Blu-ray, discrete multi-channel audio is old hat, and the format has nothing to prove.  Either way, in my opinion, the audio really really shines on these babies.

All that was left, it seemed, was to collect a few choice discs and keep on enjoying.  One of the best places to buy LD's is discountlaserdisc.com.  The guy who runs it is great to talk to and he really goes the extra mile to make sure everything he sells is in good condition.  The prices are quite good as well.  The site inventory isn't always up-to-date so if you're looking for something he doesn't have listed, just ask - he might have it.

Buying laserdiscs on eBay is a bit of a mixed bag.  Out of 20 or so films I've bought that way, about 5 were defective in some way.  Oddly most of the "sealed" discs I've bought have turned out bad - which leads me into the next important thing about Laserdiscs. This hobby is not for the faint of heart. Getting into Laserdiscs, just like vinyl, will re-acquaint you (or acquaint you for the first time) with analog format idiosyncrasies that our all-digital lives have sheltered you from. In just the handful of collecting that I've done in the past year, I've come across about every major kind of flaw of the format. When you consider that I have about 40 films altogether, that's a pretty high failure rate.

There is no error correction for the video stream. Any dirt, smudge or scratch that will refracts the laser will show up in the image.

Crosstalk

The discs are so large that the possibility of warping is much higher than CD/DVD/BD sized discs - so much so that Laserdisc players incorporate a "tilt" mechanism that adjusts the angle of the laser to try to keep it perpendicular as the disc is playing.  A small amount of warping is normal as keeping a surface area that large perfectly flat is pretty tricky, but a lot of people made the situation worse by stacking them horizontally for storage.  These things are pretty heavy and stacking more than 7 or 8 of them this way will cause the ones on the bottom to warp under the weight.  When a disc is too warped or the tilt mechanism drifts out of alignment it leads to a picture artifact called crosstalk.  Crosstalk results in herring-bone like warps in the picture, and when it's really bad, the picture will flicker and roll - sometimes even superimposing two different images.

The Dreaded "Laser Rot"

Unlike modern optical disc formats the layers of which are fused by heating the acrylic plastic until it melts together, Laserdisc layers were actually glued together.  The reason this is significant is that if the disc wasn't manufactured properly, over time the glue will lose its seal, allowing tiny amounts of oxygen to leak into the edges of the disc and oxidize the aluminum reflective layer where the information is etched onto the disc.  This creates little bubbles in the track that refract the laser and prevent it from properly reading the information.  The result comes out in the picture as bright speckles or dots.  This is known as "laser rot" and among LD enthusiasts is like the bubonic plague.  The worst part of it is that it's a progressive degeneration.  A disc can play fine once, but a year later be so degraded it can't be read at all.

Contrary to superstition (yes, there is a lot of actual superstition about laser rot), not all bright speckles or dots are actually laser rot.  Laser rot is characterized by being worse at the beginning or the end of a disc side (where the physical edges of the disc are) and usually persists for several seconds to several minutes and clears up the farther the laser is from the edges of the disc.  The occasional random bright dot is not laser rot.  It's usually a manufacturing impurity or a spec of dirt on the disc surface.  The reason it's important to distinguish the two is that rot will usually get worse, but the occasional bright dot will not.


Hedging Your Bets

In perusing forms and ebay auctions, I didn't immediately understand why so many people kept 5 or 6 players around at any given point.

After about 3 months of light use, my shiny new LD player broke down.  Upon seeking support, I was surprised by how much information I was able to find online about troubleshooting and repair.  The Laserdisc collector community is an unusual to say the least.  The heyday of Laserdisc enthusiasm was largely pre-internet, and a lot of the information I was able to track down came from mid-90's style web pages with static html, blinking gifs and visitor counters.  For some players, service and user manuals are available freely to download, and others are hoarded like the last box of twinkies in a post-apocalyptic wasteland.  And there are still a ton of good places to go to get advice like the forums at lddb.com and www.vintage-radio.net. 

I couldn't find a service manual for my particular model of player, but I did find one that was just a couple of model numbers off and it gave me some decent clues about what to look for and a general idea of how the player worked.  As annoying as it was not to be able to watch all of the discs I had been buying up, the process of trying to fix the player was actually quite cathartic. At this point I've torn the thing down and put it back together so many times I could probably do it blindfolded in the dark.  In all I ended up buying 3 players in various states of disrepair, and from three broken players I ended up with 2 working units.  With time, patience and not a small amount of prayer, I might eventually fix the third player as well.

The Previous Frontier

It's probably been said already -  this is a rewarding hobby just not one for the faint of heart.  In my lifetime I've purchased about 7 Laserdisc players via Ebay, and only two of them arrived in working condition.  (I've so far been able to repair all but one of them). Out of 50 films I've bought three had laser rot to various extents (all were perfectly watchable despite) and three had been ruined by being stacked horizontally for too long.  Despite the difficulty, working on the players and watching the films in this archaic but surprisingly good format has been some of the most fun I've ever had watching movies at home.  If you don't mind working for it, and don't have to see every movie in the highest fidelity possible, definitely consider dipping your toe into the retro world of Laserdisc.








Sunday, May 01, 2016

Vash

My Dearest Vash,

From the time I first met you, I've loved you deeply.  Before you we barely left the house.  But before we knew it we were going out for play dates and puppy playtime at the pet store. You were the apple of your adoptive mother's eye, bringing her joy and brightening our family with a light that will never fade though it shines no more.  That you loved her could not be denied.  You carried your burden with seldom a complaint, always eager and dutiful, bright-eyed and optimistic.  I will never forget being greeted every morning by a puppy who was so overjoyed to see me that he could hardly contain the emotion in his body so all of him wagged.  Your paws were so huge.  I had to start covering sensitive parts with my hands to keep you from punching me as you raised your paw in friendly welcome, though on the couple of occasions when I forgot and your "fist" connected, your mother laughed and you thought you had pleased me greatly - why else would I have bent over to give you access to lick my face?  We had such good times, I don't want to believe that they're over.

I made so many mistakes with you.  But you always loved me, and when I finally realized what I was doing wrong, you were still there, ever patient and willing to love me like it had never happened.  Thank you so much for not holding me accountable for the mistakes I made discovering how to be a good father.  Thank you so much for just being you.

Selfishly I had hoped that when the end came it would be sudden or inescapable, but life sometimes presents us with hard choices.  They wouldn't be hard if they were clear or inescapable.  No matter how necessary or merciful my decision was, I may never forgive myself for choosing to end your life.  What a wretched creature I am to believe that I have the wisdom to decide such things, and what a coward to wish against the responsibility of doing so anyway.  If you could have understood what was happening, not a shred of doubt resides within me that you would have loved me anyway and agreed cheerfully.  Perhaps it is an inadequate monument to my love for you, but the decision has set my heart and mind at war with one another.  Though I hate myself for it, I loved you too much to choose any other path.  But enough about me.

You were the most beautiful creature, at once elegant and fearsome, regal and rough. People used to cross the street to tell us how beautiful you were, and cross the street to stay out of our path.  I once had a random man walk up to me in the parking lot of a supermarket and offer me $500 for you.  No doubt he believed it was a king's ransom.  I didn't have the heart to tell him what we had paid for you.

You were on the news once.  A local animal ordinance was being hotly debated by the city council and a reporter thought you would make for a great clip so she stopped and asked me a bunch of questions, while they got some shots of you.

Your paws were so enormous that we thought you were going to be a giant, but you turned out to be the perfect size for your breed.

How much you loved playing tug-of-war with your rope, and catching full-sized soccer balls in your enormous jaw.  Your kill count for both of them had to be in the dozens.

Your bark was so resonant and fearsome, I once saw a man jump nearly two feet straight up in the air in reaction to it.  But you were such a gentle soul for your fearsome appearance - you never killed a single living thing.

You were there to greet each new addition to our family cheerfully and warmly.  You never complained or acted out as your adoptive parents' attention divided again and again and your portion of it diminished.  It took you the longest time to accept that not every new guest was a permanent resident - how hard you cried when they would leave at the end of their visit, and how jubilantly you recognized and welcomed them whenever they returned.

Once we heard you making an unusual and desperate whimper.  Our oldest son was just learning to walk and still leaning up against walls for stability.  You loved to lay in front of the sliding glass door, so when our son went to look out the window you were under foot.  When your mother came to investigate she found him standing on your testicles, and a look of shock and desperation on your face. "Get this kid off me!!!"  We couldn't believe your patience and long suffering.

You always used your senses and intuition for the betterment of the pack, howling at tornado sirens that weren't even audible to us.  When your adoptive mother was with child you were doubly vigilant and insisted on being by her side.

When you first came to stay with us, the snow was piled so high your mother had to dig a trench through the back yard just so you could do your business.  When the spring came and the ground began to thaw, I had a hard time getting you to even walk on the grass, and even when you did, you always sought out snow piles to do your business on.

You loved to play ball so much.  Once we went to the lake in early spring when the surface was still frozen over, and somehow your ball ended up out on the ice.  It was everything we could do to keep you from going after it.  You cried so desperately for it, that I eventually slid out onto the ice on my belly to get the ball back for you.

Your paws always smelled of freshly popped popcorn for some reason, so one of your many nicknames was "Popcorn Paws".

Your favorite place was always the bathtub.  I found you there so often that I used to call you "Tub Puppy".

When you were about a year old, we took you to Colorado with us.  How you loved it there.  Denver was such a dog-friendly town that we were invited everywhere.  I think your favorite was the leather goods store.

You were always so enthusiastic, sometimes to the point of silliness that I used to make up silly little songs while I rubbed your snout or scratched behind your ears.  Who is the foofer doodle dog? Vash is.

You most generous gift to us has been the memories which we will hold onto with all that is in us.  The hole in my heart is painful, but I wouldn't trade it for anything because it represents the privilege of having known you. You carried your burden with unwavering constancy. No one will ever carry it the way you did, and we will always wish we had more time with you, but rest easy, my friend, you burned so brightly and left this world a better place for having been here.

Goodbye, Puppers.