Tuesday, January 31, 2017

New Nintendo 3DS XL motherboard swap

The Super Famicom is a contender for my favorite game console of all time.  I loved the North American redesign as the Super NES, but the Super Famicom was what I fixated on for years leading up to its release.  Everything from the contours of the console to the colors of the buttons on the controller seemed so lovingly and carefully designed.  Nothing before or since has really come close.

So when Nintendo announced in early 2016 that there would be a limited edition New Nintendo 3DS LL (that's what the XL is called in Japan) to celebrate the anniversary of the Super Famicom, I immediately pre-ordered it from National Console Support, then dutifully waited 8 months until it arrived.  Whenever a new 3DS special edition is announced in Japan, there's usually a bit of uncertainty about whether it will also come out in the U.S., but this one was clearly not going to be brought here.  It's doubtful most Americans would even recognize what it was supposed to be.

When it arrived, I was thrilled.  Even the product box was a callback to the original Super Famicom box, with the simple stylized stencil design and gray background. 

But enough about that, the N3DS Super Famicom edition is awesome, etc.

Problem is that Nintendo, in a colossally frustrating and reviled move decided that the 3DS would be their first handheld ever to be region locked, so basically I had this cool handheld that would only play Japanese games.  I don't actually own any Japanese 3DS games, and what I really want is to play my US games on this system.

In olden times when we wanted to get around a region lock, we bought and installed mod chips or used boot disks, and more recently used softmods.  Unfortunately Nintendo did a really thorough job of embedding the region lock.  There are, apparently, a couple of region bypass methods for the 3DS, but they all require that you be on some old version of firmware and never update your system, and/or that you have a copy of some obscure game like Cubic Ninja which is only expensive because people want it to help trigger an exploit.  These hacks are involved, not permanent, and always in danger of getting patched out in a future firmware release.  What's most ridiculous about all this is the fact that even flash carts which allow you to play illegally copied games all day long are STILL stymied by region locks.

So the next best thing is to put a U.S. brain (a.k.a. motherboard) into a Japanese 3DS.  When the New 3DS XL was brand new this was a pretty expensive proposition as the only way to get such a motherboard was to buy a domestic New 3DS XL.  However after a year or two a lot of these have been damaged to various degrees and you can get a broken one much more cheaply.  In my case a repair shop was parting out broken units and I was able to get a working motherboard for just over the price of a new game.

The process of transplantation is straightforward but a little tedious. There are plenty of guides online about teardown, and even some videos showing disassembly and reassembly but they're all very terse and leave out little nuances.

There are over a dozen delicate little connectors that need to be carefully removed before the old motherboard will come out and almost all of them are attached to flimsy little ribbon cables.  Every instruction I've ever read that involved removing or replacing ribbon cables came with a stern warning about how easy they were to damage and basically made it seem like they were made of tissue paper. I've done a lot of repairs where I had to twist or pull on the ribbon cables to get them into position, and in a couple of instances I've accidentally pulled too hard or twisted one badly enough I was sure it wouldn't work when everything was back together, but in every case they were fine. I've never actually damaged one. Maybe that's because all the warnings made me extra careful?

Anyway, there are several different types of connectors and they each have to be removed a little differently.

The L and R button clusters are connected to the back of the unit with little sockets. You can pry them up easily with a fingernail or small flathead screwdriver.  The cart slot is also connected to the motherboard with a larger version of the socket connector.

The WiFi antenna (the red wire) is a connected by a standard socket - the same kind you'd see in a laptop.  It's pretty sturdy, just pry it off.


There are four kinds of ribbon connectors - I've color coded them in the image above.

The narrow all-black connectors (outlined in red) just use tension - gently pull them straight out and gently push them straight in. For the narrow black connectors with a large hinged gray lever (outlined in gray) gently pry up the gray lever from the opposite side of where the cable is inserted and the cable should just slip out. (The analog stick ribbon is folded over one of these - you'll need to unscrew the backing for the analog stick and pry it up before you'll be able to get at its ribbon connector.

The wider levers are color-coded according to which direction the connectors should be inserted. The dark sockets with light colored levers (outlined in blue) indicate the ribbon needs to be inserted contact (i.e. shiny) side up.  The light colored sockets with the dark levers (outlined in green) indicate the ribbon needs to be inserted contact side down. There are two wide levers not shown in the picture that are underneath the board in the upper-right corner. One of the two on the back side will actually fit either way - be careful and follow the color code.

The trickiest part of getting the thing apart and back together involves the 3 ribbon cables going into the hinge in the upper right (as your face the back). These ribbon cables are literally right on top of each other and have very little slack to work with.  What's worse is that they are constantly in the way of each other when you try to reassemble them.  (I had to resort to using a sliver of tape to hold one of them out of my way while I worked on the first ribbon.) If you don't have small fingers (and I very much don't) you'll definitely want a good precise pair of angled tweezers. The ribbon on the very bottom (on the underside, the longest of the three that connects perpendicular to the others) is jammed into the hinge recess at an odd angle so when it comes undone, it's not intuitive how it's supposed to go back in.  This is where the color coding for the connectors comes in really handy - it has a black hinge on the socket, so it's conductor side down (in relation to the surface of the board).  I strongly recommend taking pictures before removing the ribbons so you have a reference for how to fold them back in.

If it's not obvious from the subtext, I managed to insert one of the ribbons backwards and the result was that when I turned the system on, the power light would come on and after a second or two there would be a low pop and the system would power down.  Fortunately this didn't cause any apparent damage.  When I fixed the cable orientation everything came up normally and has been working perfectly for a couple of weeks with no issues.








Online Resizing of a Boot Partition on RHEL 6 or 7 with parted.

The Problem:

I've been getting away with using a 200MB boot partition for over a decade, but RHEL 7 has started pushing the limits of that scheme.  Between GRUB2 and the automatic creation of a rescue kernel and initramfs that's twice the size as a normal initramfs, that 200MB gets consumed much more quickly.  I can only make it through about two update cycles before having to uninstall the old kernels to avoid upgrade failures.

This is a handy command for removing all of the old kernels by the way:

yum erase $(rpm -qa kernel* | egrep -v $(uname -r))


I've updated my provisioning tools to make the boot partition larger, but obviously I still have to do something about the pre-existing systems.


All of my VMs are built with provisioning templates that create a boot partition on the first disk, then format the remainder of the first disk for LVM.  In order to expand the boot partition, it's boundary needs to be moved into space occupied by the OS itself.  Fortunately LVM is pretty flexible.  Not only can I "move" the partition boundary, I can do it while the system is up and running.

Here's what we're starting off with:

[root@azrst2ill001 ~]# parted -s /dev/sda print
Model: Msft Virtual Disk (scsi)
Disk /dev/sda: 53.7GB
Sector size (logical/physical): 512B/4096B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  211MB   210MB   primary  xfs          boot
 2      211MB   53.7GB  53.5GB  primary               lvm


Two partitions on the first disk, the first partition is for /boot, the second is owned by LVM. As you can see there's no space between them: the second partition begins exactly where the first ends.


Here's what LVM looks like:

[root@azrst2ill001 ~]# vgs
  VG        #PV #LV #SN Attr   VSize   VFree
  rootvg      1   6   0 wz--n-  49.80g 21.94g

[root@azrst2ill001 ~]# pvs
  PV         VG        Fmt  Attr PSize   PFree
  /dev/sda2  rootvg    lvm2 a--   49.80g 21.94g


The primary volume group with all of the OS volumes is named "rootvg" and it is contained on a single PV - /dev/sda2.  With some quick mental math you can see it's consuming just over 30GB.

Note:These instructions assume the machine was built with the default filesystem for each release, so the RHEL 6 instructions assume you built /boot with ext4, otherwise the instructions assume you're working with RHEL 7 and built /boot with xfs.

Warning:
This process is somewhat dangerous because if it is done incorrectly or the server is rebooted before it is complete, the system could be unbootable..


The Goal:
Expand the boot partition from 200MB to 1GB without re-imaging the system.


The Solution:


1. Verify the volume group has at least 1GB free because that's basically where the extra space for /boot is going to come from.

vgs


2. Add a temporary disk to the machine that's at least as large as the root volume group.  In my case I attached a new virtual data disk in Azure and 128GB was the smallest option.

UDEV created the device as /dev/sdc:

[root@azrst2ill001 boot]# lsblk
NAME                          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                             8:0    0   50G  0 disk
├─sda1                          8:1    0  953M  0 part /boot
└─sda2                          8:2    0 49.1G  0 part
  ├─rootvg-rootlv (dm-0)      253:0    0    8G  0 lvm  /
  ├─rootvg-localhomelv (dm-1) 253:1    0    2G  0 lvm  /usr/local/home
  ├─rootvg-optlv (dm-2)       253:2    0    5G  0 lvm  /opt
  ├─rootvg-swaplv (dm-3)      253:3    0    2G  0 lvm
  ├─rootvg-tmplv (dm-4)       253:4    0    5G  0 lvm  /tmp
  └─rootvg-varlv (dm-5)       253:5    0    4G  0 lvm  /var
sdc                             8:32   0  128G  0 disk
sdb                             8:16   0   56G  0 disk
└─sdb1                          8:17   0   56G  0 part /mnt/resource




3. Add the new disk to the primary VG

vgextend rootvg /dev/sdc


4. Move all of the physical extents (the data basically) off of /dev/sda2. This process will take some time.

pvmove /dev/sda2 


5. Once all of the data is cleared off of /dev/sda2, delete it from the primary VG.

vgreduce rootvg /dev/sda2


6. Just for good measure remove /dev/sda2 from LVM altogether

pvremove /dev/sda2


7. (Optional) At this point I wanted to make sure I had some kind of fallback if I dorked something up, so I made a couple of quick backups.

Firstly, I backed up the whole boot partition:

dd if=/dev/sda1 of=/root/bootpart.img

Next I backed up the MBR

dd if=/dev/sda of=/root/mbr.img bs=512 count=1

These backups are only really helpful if you realize you screwed something up before trying to reboot.


8. Delete the LVM partition

parted -s /dev/sda rm 2


9. Put the starting position of the boot partition into a variable

START1=$(parted -s /dev/sda print | grep "^ 1" | awk '{print $2}')


10. (The scary part) remove the boot partition.  

You can get away with this because the only thing parted is manipulating is the partition table.  The actual filesystem data is unaffected by this.  As long as you put the start of the boot partition in exactly the same place (which is why we saved that place in a variable before deleting it) everything will line up and work normally.

umount /boot
parted -s /dev/sda rm 1


11. Re-create the boot partition

parted -s /dev/sda mkpart primary xfs $START1 1GB

If you're doing this on RHEL 6, you probably want to go with ext4 instead:

parted -s /dev/sda mkpart primary ext4 $START1 1GB


12. Set the partition to bootable

parted -s /dev/sda set 1 boot on


13. Check the filesystem.

xfs_repair /dev/sda1

Note: if you get an error message  "/dev/sda1 contains a mounted filesystem" just run "umount /boot" again right before you run the command. For some reason RHEL 7 kept re-mounting /boot automatically on me(autofs isn't even installed).  I didn't take the time to figure out what was causing this behavior, but it's easy enough to work around.

If you're doing this on RHEL 6:

fsck -f /dev/sda1


14. Mount the /boot filesystem (if it didn't already mount itself)

mount /boot


15. Resize the /boot filesystem to fill the new larger partition.

xfs_growfs /dev/sda1

If you're doing this on RHEL 6:

resize2fs /dev/sda1


16. I want to use the rest of the space on /dev/sda for the second partition.  To get the start and end positions of free space, I used parted to set some variables.

START2=$(parted -s /dev/sda print free | grep Free | tail -1 | awk '{print $1}')
END2=$(parted -s /dev/sda print free | grep Free | tail -1 | awk '{print $2}')


17. Create the new LVM partition (the -- tells parted to accept a blank value for filesystem because we're going to use the partition for LVM instead of installing a filesystem onto it.)

parted -s /dev/sda mkpart primary -- $START2 $END2


18. Set the partition flag to LVM

parted -s /dev/sda set 2 lvm on


19. Check to make sure everything looks right - most importantly that the boot partition is showing the desired new size, and that it's set to bootable, and that when you do an ls on /boot you can see the kernel and intramfs files.


[root@azrst2ill001 ~]# parted -s /dev/sda print
Model: Msft Virtual Disk (scsi)
Disk /dev/sda: 53.7GB
Sector size (logical/physical): 512B/4096B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  1075MB  1074MB  primary  xfs          boot
 2      1075MB  53.7GB  52.6GB  primary               lvm



[root@azrst2ill001 ~]# ls /boot
config-3.10.0-327.10.1.el7.x86_64                        initrd-plymouth.img
config-3.10.0-327.28.3.el7.x86_64                        symvers-3.10.0-327.10.1.el7.x86_64.gz
grub2                                                    symvers-3.10.0-327.28.3.el7.x86_64.gz
initramfs-0-rescue-edcc5bfcc87f4b90a0ae36219b0138e3.img  System.map-3.10.0-327.10.1.el7.x86_64
initramfs-3.10.0-327.10.1.el7.x86_64.img                 System.map-3.10.0-327.28.3.el7.x86_64
initramfs-3.10.0-327.10.1.el7.x86_64kdump.img            vmlinuz-0-rescue-edcc5bfcc87f4b90a0ae36219b0138e3
initramfs-3.10.0-327.28.3.el7.x86_64.img                 vmlinuz-3.10.0-327.10.1.el7.x86_64
initramfs-3.10.0-327.28.3.el7.x86_64kdump.img            vmlinuz-3.10.0-327.28.3.el7.x86_64



20. Move the rootvg volume group back to the first disk, and remove the temporary disk from the VG.

vgextend rootvg /dev/sda2
pvmove /dev/sdc
vgreduce rootvg /dev/sdc
pvremove /dev/sdc


At this point it should be safe to remove the temporary disk you added in step 2, and to delete the backup .img files if you chose to create them in step 7.


Even though you can perform this whole operation with no downtime to the system, I strongly recommend rebooting at this point to verify everything is working as it should.  Better to find out now than months from now when it's not fresh in your mind.