Recovering a Boot-Looping Remarkable 2 Via the POGO Pins

Attempt to recover via SSH

During a boot-loop event, it may be possible to connect for a very short period via SSH to perform a command. This requires that SSH-Keys have been setup, though there’s an option as well if the SSH password is known.

Because the window for executing a command is extremely small, executing the command within an all in one script is desirable. The following one-line command will spawn 200 threads which all attempt to connect and execute a recovery command on a loop to maximise the probability of success.

for i in {1..200}; do while true; do timeout 2 ssh root@10.11.99.1 systemctl disable xochitl 2>/dev/null; done & sleep 0.01; done

If the ssh password is known then a utility like passh can be tried instead of ssh.

Getting into Recovery Mode

Follow the instructions from the Recovery Github. Here’s an overview of the required steps.

• Power Down the Remarkable completely,. If the device is boot looping, hold the power down button for 10 seconds as it’s starting.
• On the computer run sudo dmesg -w
• Connect the USB-C breakout board to the remarkable. Connect the resistor to bridge D8 (SBU2) and B12 (GND).
• Connect the POGO adapter to the side of the remarkable on the metal contacts. Ensure correct orientation of the adapter and ensure the pins are in the correct order.
• Plug in the POGO adapter to the USB Port. Due to the danger of damaging the USB Port, it’s recommended to use an external adapter.
• Check to see if the correct messages have appeared in dmesg (… Freescale SemiConductor Inc …). If not, unplug and recheck the connections.

• Once connected, remove the pull-down resistor from the usb adapter
• Run the imx_usb binary from the recovery repository with `sudo ./imx_usb'

Once successful the remarkable screen will flash, and four new partitions will appear in the host OS available to be mounted. The partitions are:

1. Uboot Partition /var/lib/uboot
2. System Root Partition /
3. Update partition - Not Mounted normally
4. User Home Partition /home

Recovering the device

There are a few things to check:

• Check the Root Partition is not full with df -h. Even if the root partition was healthy before entering a boot loop, it may have filled up with error logs. This can be amended by mounting the root partition and removing the relevant files.
• Revert the changes you made if possible by mounting the system partition. This requires advanced knowledge of what you were attempting which caused the boot loop to fix the issue.

To mount a relevant partition, follow these steps

• Choose a suitable mount location. For this example I’ll use /mnt
• Choose the partition you wish to mount. Check by observing the dmesg logs and with lsblk where the remarkable block devices have presented themselves. For this example I’ll use /dev/sde.

Mounting the root partition:

sudo mount /dev/sde2 /mnt
# umount
sudo umount /mnt

Mounting the home partition:

sudo mount /dev/sde4 /mnt
# unmount
sudo umount /mnt

Recovering by overwriting the system files with an update

It’s possible to reflash an update file over the system files, leaving the user files in the root folder intact (xochitl files ie. all data presented during normal operation, notebooks, PDFs, ebooks etc.). This will overwrite certain configuration files for some additional installed utilities like koreader so ensure you have backed up any important information first.

Extracting the system files from an update file

The Codexctl binary will be used for downloading the desired update and extracting the necessary files from the signed raw block image.

First, download codexctl by deleting the latest relevant version from the Releases page.

mkdir codexctl
cd codexctl
wget https://github.com/Jayy001/codexctl/releases/download/1701639955/ubuntu-latest.zip
# extract
unzip ubuntu-latest.zip
# Give run permission
chmod +x codexctl.bin

Next, use the script to check and download the desired update:

./codexctl.bin list

Select from the available options the desired update. For this example I’ll proceed with using 2.15.1.1189 for the Remarkable 2.

./codexctl.bin download 2.15.1.1189

At time of writing this puts the file into the users download folder. So, grab the downloaded file and move it into the codexctl directory:

mv ~/Downloads/2.15.1.1189_reMarkable2-wVbHkgKisg-.signed .

Now codexctl can be used to extract the raw block image from the signed format:

./codexctl.bin extract 2.15.1.1189_reMarkable2-wVbHkgKisg-.signed

Now the binary can be used to mount the image:

sudo ./codexctl.bin mount extracted

This mount the image into /opt/remarkable. Once done, to unmount simply run sudo umount /opt/remarkable.

Overwriting the system files with the update

Monitor the dmesg logs as you work to spot any issues with the connection.

Mount up the system partition:

sudo mount /dev/sde2 /mnt

Delete everything on the root partition:

sudo rm /mnt/* -r

And copy over the update files, ensuring to preserve all properties on the copy command:

sudo cp -a /opt/remarkable/* /mnt

Repeat for the update partition on the 3rd partition. I’m not certain if this step is required, but it would seem to be good practice.

Once done, unmount the remarkable

sudo umount /mnt

Reboot. If you are certain the root partition is good and the device still does not boot following this, it’s reasonable to suspect there was a write error over the unideal pogo connection (keep monitoring the dmesg logs). Reset and try again.

sudo umount /opt/remarkable

Taking a full backup

Once the system is working again, you may wish to take a full raw backup of the remarkable filesystem, in case of future issues. You can use dd to achieve this:

sudo dd bs=8M status=progress if=/dev/sde of=./full-remarkable-image

Information on the remarkable FileSystem:

The Remarkable storage partition table contains a valid MBR and no GPT. By default, the layout is:

Disk ./full.dd: 7.13 GiB, 7650410496 bytes, 14942208 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x7a2e5b8e

Device     Boot   Start      End  Sectors  Size Id Type
./full.dd1         2048    43007    40960   20M 83 Linux
./full.dd2        43008   595967   552960  270M 83 Linux
./full.dd3       595968  1148927   552960  270M 83 Linux
./full.dd4      1148928 14942207 13793280  6.6G 83 Linux

Info from blkid on the block size and filesystem used:

1-uboot.dd: SEC_TYPE="msdos" LABEL_FATBOOT="BOOT" LABEL="BOOT" UUID="04C5-BC1A" BLOCK_SIZE="512" TYPE="vfat"
2-root.dd: UUID="51834950-5d31-43ca-b736-de258355a904" BLOCK_SIZE="1024" TYPE="ext4"
3-unknown.dd: UUID="51834950-5d31-43ca-b736-de258355a904" BLOCK_SIZE="1024" TYPE="ext4"
4-home.dd: LABEL="home" UUID="9364dbcf-be4d-4a45-8c72-20052c3b7c7c" BLOCK_SIZE="4096" TYPE="ext4"

Wiki Hard Reset Method

From the old wiki

How to perform a Hard Reset?

Press and hold power button for 10 seconds
Release
Press and hold power button for another 2 seconds
Release

Closing Thoughts

• There doesn’t seem to be a viable method of restoring the uboot partition. This partition is mounted with write permissions during runtime, so it’s possible that it could become damaged during use. Once a user has made their own backup then they have a source of truth to work from in the future. Sharing the missing files with other users may infringe on Remarkable intellectual property rights.
• The default root user home partition doesn’t seem to be provided anywhere, as this contains a few defaults which are lost otherwise (.bashrc). Update - this does seem to carry forward from /etc/skel with certain files being in certain versions. It’s strange the default .vimrc seems to be for vim with gui enabled, whereas the bundled vim does not include the gui libs.
• The data storage appears to utilise no encryption, and any user security lock configured seems to be by-passable by the diagnostic port.

There is a repository containing Uboot however it appears to be for the Remarkable 1 only and not RM2 (zero-gravitas not zero-sugar).

The source for Remarkable version of linux is located here.

Further work that could be done here would be:

• Further investigate the boot-looping and identify what process is responsible for detecting the failed boot and causing a reboot. Investigate disabling this process to allow for safer experimentation.
• Build a custom jig to allow for easier access to the POGO pins, rendering the backups procedure much simpler.
• Run the emulated version of remarkable to perform experiments without risk of device damage.