UBI/ubifs problem

[Ricard Wanderlof]

I posted this query yesterday, but the information I provided was rather
short, so here's another attempt with more info which hopefully will
trigger some response.

I have two ubifs images made with mkfs.ubifs:

-rw-rw-r-- 1 ricardw users 20256768 2012-02-16 11:12 rootfs.img
-rw-rw-r-- 1 ricardw users 2064384 2012-02-16 11:09 rwfs.img

On my target system I attach mtd5 to UBI and create two volumes:

ubiattach -m 5 /dev/ubi_ctrl
ubimkvol -N rootfs -n 4 -t static -s 25165824 /dev/ubi0
ubimkvol -N rwfs -n 5 -t dynamic -S 30 /dev/ubi0

30 LEBs in the flash I'm using is about 30 * 126 KiB = 3.7 MiB .

I then write the volumes with the image data

ubiupdatevol /dev/ubi0_4 ./rootfs.img
ubiupdatevol /dev/ubi0_5 ./rwfs.img

Now I mount ubi0:rootfs on /mnt/1:

mount -t ubifs ubi0:rootfs /mnt/1/

which results in

UBIFS: mounted UBI device 0, volume 4, name "rootfs"
UBIFS: mounted read-only
UBIFS: file system size: 23869440 bytes (23310 KiB, 22 MiB, 185 LEBs)
UBIFS: journal size: 9033728 bytes (8822 KiB, 8 MiB, 71 LEBs)
UBIFS: media format: w4/r0 (latest is w4/r0)
UBIFS: default compressor: lzo
UBIFS: reserved for root: 0 bytes (0 KiB)

All is well so far. Now I attempt to mount ubi0:rwfs on /mnt/2:

mount -t ubifs ubi0:rwfs /mnt/2

which results in

UBIFS error (pid 2965): validate_sb: bad superblock, error 8
mount: Mounting ubi0:rwfs on /mnt/2 failed: Invalid argument

If I now remove the rwfs volume using ubirmvol, and create a new one that
is 78 LEBs in size (smaller sizes yield the same error message), again
ubiupdatevol the same rwfs.img image, and attempt to mount it, I get:

UBIFS: reserved for root: 0 bytes (0 KiB)
UBIFS: mounted UBI device 0, volume 5, name "rwfs"
UBIFS: file system size: 8644608 bytes (8442 KiB, 8 MiB, 67 LEBs)
UBIFS: journal size: 9033728 bytes (8822 KiB, 8 MiB, 71 LEBs)
UBIFS: media format: w4/r0 (latest is w4/r0)
UBIFS: default compressor: lzo
UBIFS: reserved for root: 0 bytes (0 KiB)

The thing that really puzzles me here is that the 2 MiB ubifs image shows
up as an 8 MiB 'file system size' when mounted. It does explain why 78
LEBs are required though.

Furthermore, df says:

ubi0:rootfs 21212 17652 3560 83% /mnt/1
ubi0:rwfs 7228 432 6796 6% /mnt/2

which correlates with the statistics from ubifs above, but it seems odd
that I can't create a volume with more than 6% utilization. If I start to
fill the volume with files after mounting, I can get up to 100% use.

At any rate, the error message is odd, especially given that ubiupdatevol
didn't report any problems.

I've also tried using a static volume for rwfs, with the same results (and
of course I can't write to it so I can't put more files on it in that
case).

Even considering the overhead that the rootfs volume appears to need
(image size is 20256768 and ubifs reports a 'file system size' of
23869440), i.e. an overhead of 3612672 bytes), it doesn't add up; adding
3612672 to the rwfs.img image size of 2064384 would result in a total size
of 5677056, not 8644608.

Could there be compressed files in the rwfs.img image that somehow need to
be expanded in the flash?

It's not that we're pressed for space and need the size, but I'd like to
understand what is going on. I don't really think this is a bug, more
something that I don't understand. I can't find anything on the mtd
documentation pages that seem to relate to this though.

For the record, I'm using Linux 2.6.35 .

/Ricard

[Ricard Wanderlof]

Yesterday I reported a problem I had when flashing a ubifs volume.

Post by Ricard Wanderlof
...
UBIFS error (pid 2965): validate_sb: bad superblock, error 8
mount: Mounting ubi0:rwfs on /mnt/2 failed: Invalid argument
...

I think I've figured this out part way. I've been specifying a
--max-leb-count of 1000 to mkfs.ubifs, thinking that it basically just
sets a maximum size for the resulting file system. But it seems that it
also sets some sort of bounds for how much space it expects to take up in
the volume. If I reduce the --max-leb-count to the size of the target
volume, I get no errors and the file system mounts fine.

That being so, I think the resulting error message is odd, which indicates
there is a bug somewhere, probably in mkfs.ubifs ?

/Ricard

[Artem Bityutskiy]

Post by Ricard Wanderlof
Yesterday I reported a problem I had when flashing a ubifs volume.

Post by Ricard Wanderlof
...
UBIFS error (pid 2965): validate_sb: bad superblock, error 8
mount: Mounting ubi0:rwfs on /mnt/2 failed: Invalid argument
...

I think I've figured this out part way. I've been specifying a
--max-leb-count of 1000 to mkfs.ubifs, thinking that it basically just
sets a maximum size for the resulting file system. But it seems that it
also sets some sort of bounds for how much space it expects to take up in
the volume. If I reduce the --max-leb-count to the size of the target
volume, I get no errors and the file system mounts fine.

When you specify --max-leb-count=1000, but not specify the journal size,
mkfs.ubifs defines the journal size to be around 12% of that, but not
larger than 8MiB. This sets the low UBI volume size boundary.

So basically the error you get means that your UBI volume is too small
to fit the journal.

To see the journal size of your image use -v mkfs.ubifs option when you
generate it.

You can define your own journal size using the --jrn-size mkfs.ubifs
option.

Post by Ricard Wanderlof
That being so, I think the resulting error message is odd, which indicates
there is a bug somewhere, probably in mkfs.ubifs ?

I agree that the message is odd. Below is the patch which I'll push to
linux-ubifs.git as soon as git.infradead.org is available (it is down
now for the reasons I do not know). I am open to additional or different
suggestions. Thanks!

P.S. I did not even compile - test the below patch, but will push out a
tested version.

From: Artem Bityutskiy <artem.bityutskiy at linux.intel.com>
Date: Wed, 7 Mar 2012 16:29:45 +0200
Subject: [PATCH] UBIFS: improve error messages

Ricard complaints that the following error message is odd:

"UBIFS error (pid 1578): validate_sb: bad superblock, error 8"

and he is right. This patch improves the error messages a bit and makes
them more user-friendly.

Reported-by: Ricard Wanderlof <ricard.wanderlof at axis.com>
Signed-off-by: Artem Bityutskiy <artem.bityutskiy at linux.intel.com>
---
fs/ubifs/sb.c | 19 ++++++++++++++-----
1 files changed, 14 insertions(+), 5 deletions(-)

diff --git a/fs/ubifs/sb.c b/fs/ubifs/sb.c
index 6094c5a..4d4ef12 100644
--- a/fs/ubifs/sb.c
+++ b/fs/ubifs/sb.c
@@ -410,13 +410,23 @@ static int validate_sb(struct ubifs_info *c, struct ubifs_sb_node *sup)
}

if (c->main_lebs < UBIFS_MIN_MAIN_LEBS) {
- err = 7;
+ ubifs_err("too few main LEBs count %d, must be at least %d",
+ c->main_lebs, UBIFS_MIN_MAIN_LEBS);
goto failed;
}

- if (c->max_bud_bytes < (long long)c->leb_size * UBIFS_MIN_BUD_LEBS ||
- c->max_bud_bytes > (long long)c->leb_size * c->main_lebs) {
- err = 8;
+ max_bytes = (long long)c->leb_size * UBIFS_MIN_BUD_LEBS;
+ if (c->max_bud_bytes < max_bytes) {
+ ubifs_err("too small journal (%lld bytes), must be at least ""
+ %lld bytes", c->max_bud_bytes, max_bytes);
+ goto failed;
+ }
+
+ max_bytes = (long long)c->leb_size * c->main_lebs;
+ if (c->max_bud_bytes > max_bytes) {
+ ubifs_err("too large journal size (%lld bytes), only %lld bytes"
+ "available in the main area",
+ c->max_bud_bytes, max_bytes);
goto failed;
}

@@ -450,7 +460,6 @@ static int validate_sb(struct ubifs_info *c, struct ubifs_sb_node *sup)
goto failed;
}

- max_bytes = c->main_lebs * (long long)c->leb_size;
if (c->rp_size < 0 || max_bytes < c->rp_size) {
err = 14;
goto failed;

[Ricard Wanderlof]

Post by Artem Bityutskiy
When you specify --max-leb-count=1000, but not specify the journal size,
mkfs.ubifs defines the journal size to be around 12% of that, but not
larger than 8MiB. This sets the low UBI volume size boundary.
So basically the error you get means that your UBI volume is too small
to fit the journal.
To see the journal size of your image use -v mkfs.ubifs option when you
generate it.
You can define your own journal size using the --jrn-size mkfs.ubifs
option.

Is it possible to get mkfs.ubifs to create an image without specifying the
max-leb-count ? I.e. just to create an image that is as big as it has to
be. Admittedly, one would have to figure out the size of the resulting
volume afterwards, to provide an allocation size for ubimkvol or ubinize,
so perhaps it would help much.

Post by Artem Bityutskiy

Post by Ricard Wanderlof
That being so, I think the resulting error message is odd, which indicates
there is a bug somewhere, probably in mkfs.ubifs ?

I agree that the message is odd. Below is the patch which I'll push to
linux-ubifs.git as soon as git.infradead.org is available (it is down
now for the reasons I do not know). I am open to additional or different
suggestions. Thanks!
P.S. I did not even compile - test the below patch, but will push out a
tested version.

Thanks for the patch, I'll try to test it.

/Ricard

[Artem Bityutskiy]

Post by Ricard Wanderlof
Is it possible to get mkfs.ubifs to create an image without specifying the
max-leb-count ? I.e. just to create an image that is as big as it has to
be.

Has to be? :-) Not sure how to find this out. Basically the idea was
that you know how big is has to be and translate that to
--max-leb-count.

Post by Ricard Wanderlof

Post by Artem Bityutskiy
P.S. I did not even compile - test the below patch, but will push out a
tested version.

Thanks for the patch, I'll try to test it.

It broke compilation, the working one is attached.
--
Best Regards,
Artem Bityutskiy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-UBIFS-improve-error-messages.patch
Type: text/x-patch
Size: 1954 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-mtd/attachments/20120307/4ecf6c37/attachment.bin>

[Ricard Wanderlof]

Post by Artem Bityutskiy

Post by Ricard Wanderlof
Is it possible to get mkfs.ubifs to create an image without specifying the
max-leb-count ? I.e. just to create an image that is as big as it has to
be.

Has to be? :-) Not sure how to find this out. Basically the idea was
that you know how big is has to be and translate that to
--max-leb-count.

I think what I'm thinking about is that in the specialized case of ubifs
used for a read-only file system, such as a root file system (without
/etc, /var, and so on), it would be possible for mkfs.ubifs to create a
file system, and use as many LEBs as it needs for the actual data. On a
read-only file system there doesn't even have to be a journal, although I
don't know if that is possible.

In that case one could simply feed the required directory structure to
mkfs.ubifs, and it would create the file system image, and report how many
LEBs are required in the flash.

Post by Artem Bityutskiy
It broke compilation, the working one is attached.

Ok, I'll have a look at the new one instead.

/Ricard

[Ricard Wanderlof]

Post by Ricard Wanderlof
I think what I'm thinking about is that in the specialized case of ubifs
used for a read-only file system, such as a root file system (without
/etc, /var, and so on), it would be possible for mkfs.ubifs to create a
file system, and use as many LEBs as it needs for the actual data. On a
read-only file system there doesn't even have to be a journal, although I
don't know if that is possible.

Well, I'd accept a patch which added a --optimize-for-ro option which
would prepare a file-system with the smallest journal size.

Can the journal be eliminated completely or does something in ubifs
require there to be a journal at all times, albeit small?

/Ricard

[Artem Bityutskiy]

Post by Ricard Wanderlof

Post by Ricard Wanderlof
I think what I'm thinking about is that in the specialized case of ubifs
used for a read-only file system, such as a root file system (without
/etc, /var, and so on), it would be possible for mkfs.ubifs to create a
file system, and use as many LEBs as it needs for the actual data. On a
read-only file system there doesn't even have to be a journal, although I
don't know if that is possible.

Well, I'd accept a patch which added a --optimize-for-ro option which
would prepare a file-system with the smallest journal size.

Can the journal be eliminated completely or does something in ubifs
require there to be a journal at all times, albeit small?

I think currently at least 5 LEBs are required:

/* Minimum number of logical eraseblocks in the log */
#define UBIFS_MIN_LOG_LEBS 2
/* Minimum number of bud logical eraseblocks (one for each head) */
#define UBIFS_MIN_BUD_LEBS 3
/* Minimum number of journal logical eraseblocks */
#define UBIFS_MIN_JNL_LEBS (UBIFS_MIN_LOG_LEBS + UBIFS_MIN_BUD_LEBS)

[Artem Bityutskiy]

Post by Ricard Wanderlof

Post by Artem Bityutskiy

Post by Ricard Wanderlof
Is it possible to get mkfs.ubifs to create an image without specifying the
max-leb-count ? I.e. just to create an image that is as big as it has to
be.

Has to be? :-) Not sure how to find this out. Basically the idea was
that you know how big is has to be and translate that to
--max-leb-count.

I think what I'm thinking about is that in the specialized case of ubifs
used for a read-only file system, such as a root file system (without
/etc, /var, and so on), it would be possible for mkfs.ubifs to create a
file system, and use as many LEBs as it needs for the actual data. On a
read-only file system there doesn't even have to be a journal, although I
don't know if that is possible.

Well, I'd accept a patch which added a --optimize-for-ro option which
would prepare a file-system with the smallest journal size.

[Paul Parsons]

Post by Ricard Wanderlof
Is it possible to get mkfs.ubifs to create an image without
specifying the max-leb-count ? I.e. just to create an image
that is as big as it has to be. Admittedly, one would have
to figure out the size of the resulting volume afterwards,
to provide an allocation size for ubimkvol or ubinize, so
perhaps it would help much.

When I first started to try out UBI/UBIFS I had problems understanding the
arithmetic, and it took several attempts to get it working.

In the end I created a simple shell script to do all the hard work for me.

Given the UBI MTD partition size, PEB size, and desired volume sizes, the
script would do the rest:

#!/bin/bash

#
# Create UBI for 125MiB MTD on NOR flash, consisting of:
#
# UBI overhead.
# Volume 0: rootfs, 75MiB less UBI overhead.
# Volume 1: homefs, 50MiB.
#

declare -r rootimg="rootfs.img"
declare -r homeimg="homefs.img"
declare -r ipaqcfg="ipaq.cfg"

declare -ir KiB=1024
declare -ir MiB=$((1024*1024))

declare -i UBI=$((125*MiB)) # UBI size
declare -i Sp=$((256*KiB)) # PEB size
declare -i P=$((UBI/Sp)) # Total number of PEBs
declare -i O=128 # EC header + VID header overhead
declare -i Sl=$((Sp-O)) # LEB size
declare -i B=0 # NOR does not have bad PEBs
declare -i U=$((((B+4)*Sp)+(O*(P-B-4)))) # UBI overhead
declare -i Ur=$(((U+Sp-1)&(~(Sp-1)))) # UBI overhead rounded up to next PEB size

declare -i ROOTFS=$(((75*MiB)-Ur)) # rootfs size
declare -i HOMEFS=$((50*MiB)) # homefs size

cmd="mkfs.ubifs --root=rootfs --output=${rootimg} -m 1 -e ${Sl} -c $((ROOTFS/Sp)) -x none"
echo ${cmd}
${cmd} || exit 1

cmd="mkfs.ubifs --root=homefs --output=${homeimg} -m 1 -e ${Sl} -c $((HOMEFS/Sp)) -x none"
echo ${cmd}
${cmd} || exit 1

rm -f ${ipaqcfg}
echo "[rootfs]" >> ${ipaqcfg}
echo "mode=ubi" >> ${ipaqcfg}
echo "image=${rootimg}" >> ${ipaqcfg}
echo "vol_id=0" >> ${ipaqcfg}
echo "vol_size=$((((ROOTFS)/Sp)*Sl))" >> ${ipaqcfg}
echo "vol_type=dynamic" >> ${ipaqcfg}
echo "vol_name=rootfs" >> ${ipaqcfg}
echo "[homefs]" >> ${ipaqcfg}
echo "mode=ubi" >> ${ipaqcfg}
echo "image=${homeimg}" >> ${ipaqcfg}
echo "vol_id=1" >> ${ipaqcfg}
echo "vol_size=$((((HOMEFS)/Sp)*Sl))" >> ${ipaqcfg}
echo "vol_type=dynamic" >> ${ipaqcfg}
echo "vol_name=homefs" >> ${ipaqcfg}

cmd="ubinize -o ipaq.ubi -p ${Sp} -m 1 -v ${ipaqcfg}"
echo ${cmd}
${cmd} || exit 1

It works for me. Hopefully it might be useful to others.