U-Boot: using nand write to flash an ubi image, ubi part fails

Discussion:

Lars Michael

2011-06-01 07:57:07 UTC

Hello,

Using mkfs.ubifs and ubinize I have created an image with 5 volumes. In Linux I could flash it using "ubiformat /dev/mtd1 -f /root/ubi.img" and attach it using "ubiattach /dev/ubi_ctrl -m 1". It worked.

In production I would prefer to write the image from U-Boot. So in U-Boot I used "nand erase" then "nand write" to flash the image. Then I want to attach by "ubi part" command - but this fails: "UBI error: ubi_init: UBI error: cannot initialize UBI, error -17"

Then I booted Linux (via nfs) and tried to re-format+attach, but now it fails. Format complains about eraseblock contents and missing erase counters:

ubiformat: warning!: 1099 of 2036 eraseblocks contain non-ubifs data
ubiformat: warning!: only 937 of 2036 eraseblocks have valid erase counter

The ubiattach now reports lots of:
...
UBI error: check_corruption: PEB 0 contains corrupted VID header, and the data does not contain all 0xFF, this may be a non-UBI PEB or a severe VID header corruption which requires manual inspection

UBI warning: process_eb: valid VID header but corrupted EC header at PEB 4
...

I realize that the U-Boot 'nand erase' does not preserve the erase counters. So what would be the correct way to erase and write in U-Boot? And why is the ubiformat not able to correct the problems?

My kernel is 2.6.29 patched with 165 patched from the ubi back port tree.
U-Boot is 2009.08 with ubifs support, but I am not sure if there are relevant patches. The CPU is a v4 ColdFire and 256MB Micron NAND.

Any help is appreciated, thanks. Below are the detailed listing:

-> mtd

device nand0 <NAND>, # parts = 2
#: name size offset mask_flags
0: u-boot 0x00100000 0x00000000 0
1: ubi 0x0ff00000 0x00100000 0

active partition: nand0,0 - (u-boot) 0x00100000 @ 0x00000000

defaults:
mtdids : nand0=NAND
mtdparts: mtdparts=NAND:1m(u-boot),-(ubi)

-> nand erase ubi

NAND erase: device 0 offset 0x100000, size 0xff00000
Erasing at 0x100000 -- 0% complete.
<cut>
Erasing at 0xfd60000 -- 99% complete.
Skipping bad block at 0x0ff80000

Skipping bad block at 0x0ffa0000

Skipping bad block at 0x0ffc0000

Skipping bad block at 0x0ffe0000

OK

Using FEC0 device
TFTP from server 10.28.151.113; our IP address is 10.28.151.112
Filename '/twr5441x/ubi.img'.
Load address: 0x40010000
Loading: *#################################################################
#################################################################
<cut>
#################################################################
########################################################
done
Bytes transferred = 58064896 (3760000 hex)
-> md 40010000 100

40010000: 55424923 01000000 00000000 00000000 UBI#............
40010010: 00000800 00001000 27162ae3 00000000 ........'.*.....
40010020: 00000000 00000000 00000000 00000000 ................
40010030: 00000000 00000000 00000000 7d780a89 ............}x..
40010040: ffffffff ffffffff ffffffff ffffffff ................
<cut>
400113f0: 00000000 00000000 00000000 00000000 ................
-> nand write ${loadaddr} 100000 ${filesize}

NAND write: device 0 offset 0x100000, size 0x3760000
58064896 bytes written: OK
-> ubi part ubi

Creating 1 MTD partitions on "nand0":
0x000000100000-0x000010000000 : "mtd=1"
UBI: attaching mtd1 to ubi0
UBI: physical eraseblock size: 131072 bytes (128 KiB)
UBI: logical eraseblock size: 126976 bytes
UBI: smallest flash I/O unit: 2048
UBI: VID header offset: 2048 (aligned 2048)
UBI: data offset: 4096
UBI warning: process_lvol: volume table copy #1 is corrupted
UBI: create volume table (copy #1)
UBI: volume table was restored
UBI: volume 5 ("datafs") re-sized from 133 to 1316 LEBs
UBI: attached mtd1 to ubi0
UBI: MTD device name: "mtd=1"
UBI: MTD device size: 255 MiB
UBI: number of good PEBs: 2036
UBI: number of bad PEBs: 4
UBI: max. allowed volumes: 128
UBI: wear-leveling threshold: 4096
UBI: number of internal volumes: 1
UBI: number of user volumes: 5
UBI: available PEBs: 0
UBI: total number of reserved PEBs: 2036
UBI: number of PEBs reserved for bad PEB handling: 20
UBI: max/mean erase counter: 5/0
UBI error: ubi_init: cannot attach mtd1
UBI: mtd1 is detached from ubi0
UBI error: ubi_init: UBI error: cannot initialize UBI, error -17
UBI init error -17
exit not allowed from main input shell.

------------------

[root at M54418TWR root]# ubiformat /dev/mtd1 -f /root/ubi.img
ubiformat: mtd1 (NAND), size 267386880 bytes (255.0 MiB), 131072 eraseblocks of 131072 bytes (128.0 KiB), min. I/O size 2048 bytes

libscan: scanning eraseblock 0 -- 0 % complete
...
libscan: scanning eraseblock 2039 -- 100 % complete
ubiformat: 303 eraseblocks have valid erase counter, mean value is 0
ubiformat: 643 eraseblocks are supposedly empty
ubiformat: bad eraseblocks: 2036, 2037, 2038, 2039
ubiformat: warning!: 1090 of 2036 eraseblocks contain non-ubifs data
ubiformat: continue? (yes/no) yes
ubiformat: warning!: only 303 of 2036 eraseblocks have valid erase counter
ubiformat: erase counter 0 will be used for all eraseblocks
ubiformat: note, arbitrary erase counter value may be specified using -e option
ubiformat: continue? (yes/no) yes
ubiformat: use erase counter 0 for all eraseblocks

ubiformat: flashing eraseblock 0 -- 0 % complete
...
ubiformat: flashing eraseblock 442 -- 100 % complete

ubiformat: formatting eraseblock 443 -- 0 % complete
...
ubiformat: formatting eraseblock 2039 -- 100 % complete

[root at M54418TWR root]# ubiattach /dev/ubi_ctrl -m 1
UBI: attaching mtd1 to ubi0

UBI: physical eraseblock size: 131072 bytes (128 KiB)

UBI: logical eraseblock size: 126976 bytes

UBI: smallest flash I/O unit: 2048

UBI: VID header offset: 2048 (aligned 2048)

UBI: data offset: 4096

UBI error: check_corruption: PEB 0 contains corrupted VID header, and the data does not contain all 0xFF, this may be a non-UBI PEB or a severe VID header corruption which requires manual inspection

UBI warning: process_eb: valid VID header but corrupted EC header at PEB 4

UBI warning: process_eb: valid VID header but corrupted EC header at PEB 7

UBI warning: process_eb: valid VID header but corrupted EC header at PEB 8

UBI warning: process_eb: valid VID header but corrupted EC header at PEB 12

UBI error: check_corruption: PEB 16 contains corrupted VID header, and the data does not contain all 0xFF, this may be a non-UBI PEB or a severe VID header corruption which requires manual inspection

UBI warning: process_eb: valid VID header but corrupted EC header at PEB 22

...

Thanks,

- Lars

Artem Bityutskiy

2011-06-01 11:56:15 UTC

Permalink

OK.

Post by Lars Michael
In production I would prefer to write the image from U-Boot. So in
U-Boot I used "nand erase" then "nand write" to flash the image. Then
ubi_init: UBI error: cannot initialize UBI, error -17"

Sorry, I'm not familiar with U-boot, so probably cc-ing U-boot mailing
list would be a good idea.

Post by Lars Michael
Then I booted Linux (via nfs) and tried to re-format+attach, but now
it fails. Format complains about eraseblock contents and missing erase
ubiformat: warning!: 1099 of 2036 eraseblocks contain non-ubifs data
ubiformat: warning!: only 937 of 2036 eraseblocks have valid erase counter

OK, this means that many eraseblocks in the flash contain some data
which does not look like UBI headers.

Post by Lars Michael
...
UBI error: check_corruption: PEB 0 contains corrupted VID header, and
the data does not contain all 0xFF, this may be a non-UBI PEB or a
severe VID header corruption which requires manual inspection
UBI warning: process_eb: valid VID header but corrupted EC header at PEB 4
...

OK.

Post by Lars Michael
I realize that the U-Boot 'nand erase' does not preserve the erase
counters. So what would be the correct way to erase and write in
U-Boot?

I do not know, you need to ask U-boot-aware people. I've CCed some.

Post by Lars Michael
And why is the ubiformat not able to correct the problems?

It should be able to format any flash. Probably this is a bug. Can you
reproduce this with nandsim please? Just insmod nandsim on your PC (see
http://www.linux-mtd.infradead.org/faq/nand.html#L_nand_nandsim or
modinfo nandsim), write some garbage there or your image, and try to
format it.

If you can describe how I can reproduce this in my PC, I might look at
this. Or you can just debug ubiformat yourself.

--
Best Regards,
Artem Bityutskiy (????? ????????)

Ben Gardiner

2011-06-01 13:41:19 UTC

Permalink

Hi Lars

Post by Lars Michael
Hello,
Using mkfs.ubifs and ubinize I have created an image with 5 volumes. In Linux I could flash it using "ubiformat /dev/mtd1 -f /root/ubi.img" and attach it using "ubiattach /dev/ubi_ctrl -m 1". It worked.
In production I would prefer to write the image from U-Boot. So in U-Boot I used "nand erase" then "nand write" to flash the image. Then I want to attach by "ubi part" command - but this fails: "UBI error: ubi_init: UBI error: cannot initialize UBI, error -17"
[...]
I realize that the U-Boot 'nand erase' does not preserve the erase counters. So what would be the correct way to erase and write in U-Boot? And why is the ubiformat not able to correct the problems?

It's true that the ECs are not preserved. But it's also true that
trailing pages of 0xff are not trimmed either. You definitely need to
drop trailing 0xff pages when writing ubinized images containing UBIFS
volumes. To do this you can use either the new 'nand write' variant,
.trimffs [1] which is currently patches that have been Acked. Or you
can use the free-space-fixup feature which is available in the
ubifs-2.6 tree and requires that you use a mkfs.ubifs built from the
the mtd-utils tree.

I've never seen the particular error "ubi_init: UBI error: cannot
initialize UBI, error -17" when I have erroneously written a ubinized
image with nand write. But maybe that is due to the paritculars of my
platform drivers or it may be due to the version of u-boot you are
running there -- see below.

I reccomend giving both the 'nand write.trimffs' patches and the
free-space-fixup feature a try (indivually) and reporting back with

Post by Lars Michael
My kernel is 2.6.29 patched with 165 patched from the ubi back port tree.
U-Boot is 2009.08 with ubifs support, but I am not sure if there are relevant patches. The CPU is a v4 ColdFire and 256MB Micron NAND.

2009.08 is maybe too old. There have been several fixes to both ubi
and ubis in u-boot since then:

$git log --oneline v2009.08..u-boot/master -- fs/ubifs/ drivers/mtd/ubi/
c7b7d45 UBIFS: Fix dereferencing type-punned pointer compiler warning
64b6817 ubifs.c: BUG: Error following links
b1a14f8 UBIFS: Change ubifsload to not read beyond the requested size
cb9c09d UBIFS: Add ubifsumount command to unmount an active volume
6d8962e Switch from archive libraries to partial linking
2e5167c Replace CONFIG_RELOC_FIXUP_WORKS by CONFIG_NEEDS_MANUAL_RELOC
86af10c Fix "ubi part" cmd re-entrancy
ceeba00 UBI: initialise update marker
1b1f9a9 UBI: Ensure that "background thread" operations are really executed
6865168 ubifsmount fails due to not initialized list
87d93a1 move prototypes for gunzip() and zunzip() to common.h
4b142fe common: delete CONFIG_SYS_64BIT_VSPRINTF and CONFIG_SYS_64BIT_STRTOUL
521af04 Conditionally perform common relocation fixups
ef37c68 ubifs: Correct dereferencing of files-after-symlinks
9d7952e ubifs: Add support for looking up directory and relative symlinks
02f9990 Move __set/clear_bit from ubifs.h to bitops.h

Best Regards,
Ben Gardiner

---
Nanometrics Inc.
http://www.nanometrics.ca

Lars Michael

2011-07-18 09:29:08 UTC

Permalink

Hello Ben,

Post by Lars Michael

Post by Lars Michael
Using mkfs.ubifs and ubinize I have created an image

with 5 volumes. In Linux I could flash it using "ubiformat
/dev/mtd1 -f /root/ubi.img" and attach it using "ubiattach
/dev/ubi_ctrl -m 1". It worked.

Post by Lars Michael
In production I would prefer to write the image from

U-Boot. So in U-Boot I used "nand erase" then "nand write"
to flash the image. Then I want to attach by "ubi part"
cannot initialize UBI, error -17"

Post by Lars Michael
[...]
I realize that the U-Boot 'nand erase' does not

preserve the erase counters. So what would be the correct
way to erase and write in U-Boot? And why is the ubiformat
not able to correct the problems?
It's true that the ECs are not preserved. But it's also
true that
trailing pages of 0xff are not trimmed either. You
definitely need to
drop trailing 0xff pages when writing ubinized images
containing UBIFS
volumes. To do this you can use either the new 'nand write'
variant,
.trimffs [1] which is currently patches that have been
Acked. Or you
can use the free-space-fixup feature which is available in
the
ubifs-2.6 tree and requires that you use a mkfs.ubifs built
from the
the mtd-utils tree.

Ok, I have updated my mtd-utils and recompiled the mkfs.ubifs to get
support for the free-space-fixup feature. Next I have pulled
all ubi updates from the 2.6.29 backport tree which gave med 154 patches
including the fixup.

Now I can build the image, ubiformat, ubiattach and mount with success.
After a reboot I can ubiattach and mount with success.

Next step is to get it to work in U-Boot too.

Thanks and regards,
- Lars

Matthew L. Creech

2011-09-29 02:43:11 UTC

Permalink

Basically, this isn't a valid combination: "nand write" doesn't know
anything about UBI, and a UBIFS image isn't suitable for flashing
directly to NAND - it expects to sit on top of UBI.

There are 2 ways to get it working:

1. Don't use "nand write". If U-Boot is built with UBI support, you
can use "ubi write" to write a UBI image to flash. I _think_ you can
use this to overwrite an existing volume with a new image, preserving
ECs (haven't tried it myself). It certainly works fine to write a new
image to blank flash.

2. Use the "ubinize" tool to create an image suitable for writing
directly to NAND. This takes a UBIFS image and "wraps" it to turn it
into a UBI image, which you can then use with "nand write".

I've used both these methods to write UBIFS images to flash within
U-Boot, and the 2nd method also works with industrial NAND
programmers.

Hope this helps

--
Matthew L. Creech

Lars Michael

2011-09-30 03:44:08 UTC

Permalink

On Wed, Jun 1, 2011 at 3:57 AM, Lars

Post by Lars Michael
Using mkfs.ubifs and ubinize I have created an image

with 5 volumes. In Linux I could flash it using "ubiformat
/dev/mtd1 -f /root/ubi.img" and attach it using "ubiattach
/dev/ubi_ctrl -m 1". It worked.

Post by Lars Michael
In production I would prefer to write the image from

U-Boot. So in U-Boot I used "nand erase" then "nand write"
to flash the image. Then I want to attach by "ubi part"
cannot initialize UBI, error -17"
Basically, this isn't a valid combination: "nand write"
doesn't know
anything about UBI, and a UBIFS image isn't suitable for
flashing
directly to NAND - it expects to sit on top of UBI.

Hi Matthew,

I agree, which is why I used ubinize to create a UBI image (with 5 UBIFS images) and then tried to write that image.

1. Don't use "nand write".? If U-Boot is built with
UBI support, you
can use "ubi write" to write a UBI image to flash.? I
_think_ you can
use this to overwrite an existing volume with a new image,
preserving
ECs (haven't tried it myself).? It certainly works
fine to write a new
image to blank flash.
2. Use the "ubinize" tool to create an image suitable for
writing
directly to NAND.? This takes a UBIFS image and
"wraps" it to turn it
into a UBI image, which you can then use with "nand
write".

This is what I am doing (or plan to do). But to get it to work, I need
the space fixup feature for the nand write in U-Boot. I got that feature in Linux, so in Linux it works. Unfortunately it is not easy to patch
that one in my 2009.08 U-Boot.

Thanks for your comments.

Regards
Lars

Scott Wood

2011-10-03 20:36:39 UTC

Permalink

Post by Lars Michael

Post by Matthew L. Creech
2. Use the "ubinize" tool to create an image suitable for
writing
directly to NAND. This takes a UBIFS image and
"wraps" it to turn it
into a UBI image, which you can then use with "nand
write".

Try using a current U-Boot, and use "nand write.trimffs".

-Scott