What to do when your Raspberry Pi memory card wears out

My AI Picture Ink Display has been running for a few months and it’s been nice to see a different crazy picture appear every now and then. But recently it has taken to failing. I, of course, instantly assumed that my software was broken, but it turns out to be a hardware issue with the memory card in the Raspberry Pi. The Stable Diffusion program that makes the pictures works the file store very hard, repeatedly reading and writing data, and this can result in the storage device failing. I diagnosed the problem by taking a look at recent Linux kernel messages:

dmesg -T | grep -Ei 'error|ext4|mmc|i/o|corrupt'

The statement above filters the error messages for ones about storage issues. In this case, the important message was:

I/O error, dev mmcblk0, sector 22491848 ... (READ)

mmcblk0 is normally the Raspberry Pi's microSD card. An I/O error at this level means Linux attempted to read data from the storage device and the operation failed. The most precious thing on any computer is the data, in this case the pictures that had been generated, so while the system was still working I needed to get those files off the machine. I used a USB memory stick. I plugged it in, waited a few seconds for the Pi to notice it, and then checked to see what devices were available.

lsblk -f

I got this:

That told me that the usb memory key was working and had the path of “/media/rob/HP v100w” (note that this contains a space). You will get a slightly different path. Now I could copy the contents of the generated_images folder onto the USB device.

cp -av /home/rob/generated_images/ "/media/rob/HP v100w/"

The quotes around the destination are necessary because the USB drive's name, HP v100w, contains a space. The trailing / on /home/rob/generated_images/ means copy the contents of the directory into the destination directory. Once the copy was complete, the next thing to do was make sure that any cached writes to the usb drive were complete:

sync

Then then I could unmount the USB drive:

sudo umount "/media/rob/HP v100w"

Now I had a copy of the folder on the drive. If my Pi was running with the desktop this would have been much easier, I could have done it all in the File Explorer application. But the image generator runs “headless”. One tip: if the USB drive is not picked up when you plug it in, try using a different USB port. You should see blue and black usb sockets. I’ve found that for usb drives the black ones work best.

As for the suspect memory card; once I’ve got all my files off it the card is going in the bin. And I’m going to look into running the program on a device with a “proper” solid state disk. This will go faster and should not be susceptible to wear like this.