U-Boot and UBI

[Josh Boyer]

Hi All,

As has been stated before by various people, IBM has added some form of
UBI support to an older version of U-Boot. While we would love to be
able to forward port this to the latest version of U-Boot, we have
simply not found the time and don't see that being the case in the near
future.

So as an alternative, I have cleaned up the code slightly and pushed out
a version that applies to U-Boot 1.1.6. This way there is at least some
form of code release that people can use as a reference. I am not
pushing this for inclusion into the current U-Boot, as we are well aware
of the need for further adaptation and cleanup.

A few caveats:

- This is a quick forward port from a patched 1.1.4 tree to 1.1.6. It
compiles for the example board (PowerPC 440EP Bamboo), but that example
code may not actually work due to the differences in how flash was read
between the code bases. When moving to a newer U-Boot, I highly
recommend using the generic MTD interfaces.

- There might be slight differences in the UBI format that this code
base supports and what is in the upstream Linux kernel. Comparison of
the data structures in the current kernel would be advised if you are
forward porting.

- There is no write support for UBI in this port.

- The interfaces have two methods of getting scan information. The
first is having a bit of code that runs before U-Boot do the scan and
pass it. The second is to have a user/script use the "ubiscan" command
to have U-Boot manually perform the scan itself. The latter method is
likely more flexible for new boards.

All of that being said, we hope this code is useful as a guide. I will
certainly answer questions as I can. If for some unknown reason I find
a massive amount of spare time, I might even try to forward port it, but
that seems unlikely these days.

I would like to thank Frank Haverkamp, Thomas Gleixner, Andreas Arnez,
and Artem Bityutskiy for their work on UBI and the majority of the
U-Boot code we have.

The git tree can be found here:

git fetch git://git.infradead.org/users/jwboyer/u-boot.git ubi

http://git.infradead.org/users/jwboyer/u-boot.git?a=shortlog;h=refs/heads/ubi

josh

[Wolfgang Denk]

Dear Josh,

Post by Josh Boyer
As has been stated before by various people, IBM has added some form of
UBI support to an older version of U-Boot. While we would love to be
able to forward port this to the latest version of U-Boot, we have
simply not found the time and don't see that being the case in the near
future.
So as an alternative, I have cleaned up the code slightly and pushed out
a version that applies to U-Boot 1.1.6. This way there is at least some
form of code release that people can use as a reference. I am not
pushing this for inclusion into the current U-Boot, as we are well aware
of the need for further adaptation and cleanup.

...

Post by Josh Boyer
git fetch git://git.infradead.org/users/jwboyer/u-boot.git ubi
http://git.infradead.org/users/jwboyer/u-boot.git?a=shortlog;h=refs/heads/ubi

Thanks a lot.

I guess we will pick up this work ASAP.

If anybody else has intentions to work in this area, I'd appreciate if
you contact me so we can coordinate efforts.

Thanks again.

Best regards,

Wolfgang Denk

[Kyungmin Park]

Dear Josh,

First thank you for announcing UBI support on U-boot.

Actually the Samsung implemented the UBI support on U-boot already and
has used it internally. The big difference is the code base. It's
based on kernel UBI code. Yes it's not fit well to u-boot ecosystem so
it created the ubi wrapper for u-boot.

I saw your source codes and ubi commands. I think it supports minimal
features of UBI, no volume create, remove, read/write and so on like
these.

u-boot $ help ubi
ubi info [l[ayout]] - Display volume and ubi layout information
ubi create[vol] volume [size] [type] - create volume name with size
ubi write[vol] address volume size - Write volume from address with size
ubi read[vol] address volume [size] - Read volume to address with size
ubi remove[vol] volume - Remove volume
[Legends]
volume: character name
size: KiB, MiB, GiB, and bytes
type: s[tatic] or d[ynamic] (default=dynamic)

I want to know what's the better way if there are two code bases.

Please give any options and comments

Thank you,
Kyungmin Park

P.S., we will post the patches ASAP

Post by Wolfgang Denk

Post by Josh Boyer
As has been stated before by various people, IBM has added some form of
UBI support to an older version of U-Boot. While we would love to be
able to forward port this to the latest version of U-Boot, we have
simply not found the time and don't see that being the case in the near
future.
So as an alternative, I have cleaned up the code slightly and pushed out
a version that applies to U-Boot 1.1.6. This way there is at least some
form of code release that people can use as a reference. I am not
pushing this for inclusion into the current U-Boot, as we are well aware
of the need for further adaptation and cleanup.

...
Thanks a lot.
I guess we will pick up this work ASAP.
If anybody else has intentions to work in this area, I'd appreciate if
you contact me so we can coordinate efforts.

[Wolfgang Denk]

Dear Kyungmin Park,

Post by Kyungmin Park
Actually the Samsung implemented the UBI support on U-boot already and
has used it internally. The big difference is the code base. It's

The big problem is always that such ports are done completely
internally, and nothing is given back to the community. This causes a
lot of duplicated (and thus avoidavle) efforts. Samsung is one of the
companies hich have a notoriously bad record on this :-(

Post by Kyungmin Park
based on kernel UBI code. Yes it's not fit well to u-boot ecosystem so
it created the ubi wrapper for u-boot.

It would have been nice if such issued whould have been discussed
brfore implementing the stuff behind closed doors.

Post by Kyungmin Park
I saw your source codes and ubi commands. I think it supports minimal
features of UBI, no volume create, remove, read/write and so on like
these.
u-boot $ help ubi
ubi info [l[ayout]] - Display volume and ubi layout information
ubi create[vol] volume [size] [type] - create volume name with size
ubi write[vol] address volume size - Write volume from address with size
ubi read[vol] address volume [size] - Read volume to address with size
ubi remove[vol] volume - Remove volume
[Legends]
volume: character name
size: KiB, MiB, GiB, and bytes
type: s[tatic] or d[ynamic] (default=dynamic)
I want to know what's the better way if there are two code bases.

The first thing that should be done is publishing the Samsung code
without further delay. Even if it is against an old code base.

Post by Kyungmin Park
P.S., we will post the patches ASAP

Please do this *now*.


Thanks.

Best regards,

Wolfgang Denk

[Josh Boyer]

On Mon, 11 Aug 2008 14:18:53 +0900

Post by Wolfgang Denk
Dear Josh,
First thank you for announcing UBI support on U-boot.
Actually the Samsung implemented the UBI support on U-boot already and
has used it internally. The big difference is the code base. It's
based on kernel UBI code. Yes it's not fit well to u-boot ecosystem so
it created the ubi wrapper for u-boot.

Interesting.

Post by Wolfgang Denk
I saw your source codes and ubi commands. I think it supports minimal
features of UBI, no volume create, remove, read/write and so on like
these.

That is correct. Simple scan and static volume read support are
present in the code base I pushed out. That was all the project we
were doing really required in U-Boot at the time.

Post by Wolfgang Denk
u-boot $ help ubi
ubi info [l[ayout]] - Display volume and ubi layout information
ubi create[vol] volume [size] [type] - create volume name with size
ubi write[vol] address volume size - Write volume from address with size
ubi read[vol] address volume [size] - Read volume to address with size
ubi remove[vol] volume - Remove volume
[Legends]
volume: character name
size: KiB, MiB, GiB, and bytes
type: s[tatic] or d[ynamic] (default=dynamic)
I want to know what's the better way if there are two code bases.
Please give any options and comments

If your code base is more feature complete, it is certainly worth
looking at. Particularly if it is for a newer version of U-Boot.

josh

[Stefan Roese]

Post by Kyungmin Park
Actually the Samsung implemented the UBI support on U-boot already and
has used it internally. The big difference is the code base. It's
based on kernel UBI code. Yes it's not fit well to u-boot ecosystem so
it created the ubi wrapper for u-boot.

And how does NAND/OneNAND booting with UBI support fit into this? I assume
that you have some size restrictions for the IPL/SPL on your platforms as
well.

<snip>

Post by Kyungmin Park
I want to know what's the better way if there are two code bases.
Please give any options and comments

As Josh already mentioned, your version might be better suited as a basis for
mainline inclusion because its more feature complete and probably is for a
newer U-Boot version.

So best would be to publish your UBI support too, then we can compare and
decide where to go.

Post by Kyungmin Park
Thank you,
Kyungmin Park
P.S., we will post the patches ASAP

Do you have a rough idea when you could publish your patches?

Thanks.

Best regards,
Stefan

=====================================================================
DENX Software Engineering GmbH, MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: [phone] Fax: [phone] Email: office at denx.de
=====================================================================

[Kyungmin Park]

Hi,

Post by Stefan Roese

Post by Kyungmin Park
Actually the Samsung implemented the UBI support on U-boot already and
has used it internally. The big difference is the code base. It's
based on kernel UBI code. Yes it's not fit well to u-boot ecosystem so
it created the ubi wrapper for u-boot.

And how does NAND/OneNAND booting with UBI support fit into this? I assume
that you have some size restrictions for the IPL/SPL on your platforms as
well.

It's not yet covered. it's TODO

Anyway, here's UBI patch.
The real implementation was done by another board but to publish I
make a patch for apollon board for usage. (tested and verified)

As see it's all-in-one patch and contains several parts, apollon
board, OneNAND, mtdpartition, and UBI.

After the feedback, I will generate right patches for purpose.

Please give any comments and feedbacks

Thank you,
Kyungmin Park

P.S., it's bzip2ed since mailing list has size limitation.

Makefile | 1 +
board/apollon/Makefile | 2 +-
board/apollon/config.mk | 2 +-
board/apollon/ubi.c | 55 ++
common/Makefile | 1 +
common/cmd_jffs2.c | 69 ++-
common/cmd_ubi.c | 532 ++++++++++++
drivers/mtd/Makefile | 1 +
drivers/mtd/mtdcore.c | 140 +++
drivers/mtd/mtdpart.c | 531 ++++++++++++
drivers/mtd/nand/nand_base.c | 2 +
drivers/mtd/onenand/onenand_base.c | 98 ++-
drivers/mtd/ubi/Makefile | 48 +
drivers/mtd/ubi/build.c | 1182 +++++++++++++++++++++++++
drivers/mtd/ubi/cdev.c | 819 ++++++++++++++++++
drivers/mtd/ubi/crc32.c | 510 +++++++++++
drivers/mtd/ubi/crc32defs.h | 32 +
drivers/mtd/ubi/crc32table.h | 136 +++
drivers/mtd/ubi/debug.c | 192 +++++
drivers/mtd/ubi/debug.h | 152 ++++
drivers/mtd/ubi/eba.c | 1256 +++++++++++++++++++++++++++
drivers/mtd/ubi/gluebi.c | 347 ++++++++
drivers/mtd/ubi/io.c | 1274 +++++++++++++++++++++++++++
drivers/mtd/ubi/kapi.c | 638 ++++++++++++++
drivers/mtd/ubi/misc.c | 106 +++
drivers/mtd/ubi/scan.c | 1361 +++++++++++++++++++++++++++++
drivers/mtd/ubi/scan.h | 165 ++++
drivers/mtd/ubi/ubi-media.h | 372 ++++++++
drivers/mtd/ubi/ubi.h | 641 ++++++++++++++
drivers/mtd/ubi/upd.c | 441 ++++++++++
drivers/mtd/ubi/vmt.c | 862 +++++++++++++++++++
drivers/mtd/ubi/vtbl.c | 837 ++++++++++++++++++
drivers/mtd/ubi/wl.c | 1670 ++++++++++++++++++++++++++++++++++++
fs/jffs2/jffs2_1pass.c | 118 +++
include/configs/apollon.h | 21 +-
include/jffs2/load_kernel.h | 11 +-
include/linux/crc32.h | 27 +
include/linux/mtd/compat.h | 5 +
include/linux/mtd/mtd-abi.h | 1 +
include/linux/mtd/mtd.h | 27 +
include/linux/mtd/nand.h | 14 -
include/linux/mtd/nand_legacy.h | 12 -
include/linux/mtd/onenand.h | 19 +-
include/linux/mtd/partitions.h | 84 ++
include/linux/mtd/ubi.h | 186 ++++
include/linux/rbtree.h | 161 ++++
include/linux/types.h | 24 +
include/mtd/ubi-header.h | 372 ++++++++
include/mtd/ubi-user.h | 268 ++++++
include/mtd_uboot.h | 7 +
include/onenand_uboot.h | 11 +-
include/ubi_uboot.h | 222 +++++
lib_generic/Makefile | 1 +
lib_generic/rbtree.c | 397 +++++++++
54 files changed, 16383 insertions(+), 80 deletions(-)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ubi-uboot.patch.bz2
Type: application/x-bzip2
Size: 89437 bytes
Desc: not available
Url : http://lists.infradead.org/pipermail/linux-mtd/attachments/20080812/87759f61/attachment-0001.bz2

[Stefan Roese]

Post by Kyungmin Park

Post by Stefan Roese

Post by Kyungmin Park
Actually the Samsung implemented the UBI support on U-boot already and
has used it internally. The big difference is the code base. It's
based on kernel UBI code. Yes it's not fit well to u-boot ecosystem so
it created the ubi wrapper for u-boot.

And how does NAND/OneNAND booting with UBI support fit into this? I
assume that you have some size restrictions for the IPL/SPL on your
platforms as well.

It's not yet covered. it's TODO

I see.

Post by Kyungmin Park
Anyway, here's UBI patch.
The real implementation was done by another board but to publish I
make a patch for apollon board for usage. (tested and verified)
As see it's all-in-one patch and contains several parts, apollon
board, OneNAND, mtdpartition, and UBI.

Thanks. Really appreciated.

Post by Kyungmin Park
After the feedback, I will generate right patches for purpose.
Please give any comments and feedbacks

Will come... :)

Best regards,
Stefan

=====================================================================
DENX Software Engineering GmbH, MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: [phone] Fax: [phone] Email: office at denx.de
=====================================================================

[Kyungmin Park]

Hi,

Post by Stefan Roese

Post by Kyungmin Park

Post by Stefan Roese

Post by Kyungmin Park
Actually the Samsung implemented the UBI support on U-boot already and
has used it internally. The big difference is the code base. It's
based on kernel UBI code. Yes it's not fit well to u-boot ecosystem so
it created the ubi wrapper for u-boot.

And how does NAND/OneNAND booting with UBI support fit into this? I
assume that you have some size restrictions for the IPL/SPL on your
platforms as well.

It's not yet covered. it's TODO

I see.

Yes, we should make a u-boot within one block size since flash only
guarantees the first one block as bad block free. e.g., exactly
(128KiB - 2KiB) size if OneNAND case.
For this, we need to code or size optimization.
If this is solved, we can use flash as bad block free device with UBI.

Also there are some issues related with ubi partitioning.

It's just rough ideas. For make a system as reasonable it creates two
UBI devices, one for boot-loader and another for kernel. If we make
only one UBI device, we should read whole flash blocks to read kernel.
but it's not reasonable.

1. UBI image for bootloader.
boot params (rw), kernel(ro), [Optional] initrd (ro)

Even though it has r/w area, we regards it as static volume for
performance since boot params almost read only. I think If UBI has
only static volumes we can optimize it more. we don't need to mirror
layout volume. no wear leveling and so on. For this, of course, we
need special handling for boot params.

2. UBI image for kernel.
rootfs(ro/rw), datafs(rw), and others(??)

Feel free, you make it for your purposes.

Thank you,
Kyungmin Park

[Josh Boyer]

Post by Kyungmin Park
Hi,

Post by Stefan Roese

Post by Kyungmin Park

Post by Stefan Roese

Post by Kyungmin Park
Actually the Samsung implemented the UBI support on U-boot already and
has used it internally. The big difference is the code base. It's
based on kernel UBI code. Yes it's not fit well to u-boot ecosystem so
it created the ubi wrapper for u-boot.

And how does NAND/OneNAND booting with UBI support fit into this? I
assume that you have some size restrictions for the IPL/SPL on your
platforms as well.

It's not yet covered. it's TODO

I see.

Yes, we should make a u-boot within one block size since flash only
guarantees the first one block as bad block free. e.g., exactly
(128KiB - 2KiB) size if OneNAND case.
For this, we need to code or size optimization.
If this is solved, we can use flash as bad block free device with UBI.

An alternative to that is to have a very small IPL that loads U-Boot to
DRAM as the SPL. That would require some relocatable support in U-Boot.

josh

[Enrico Scholz]

Post by Josh Boyer

Post by Kyungmin Park
Yes, we should make a u-boot within one block size since flash
only guarantees the first one block as bad block free. e.g.,
exactly (128KiB - 2KiB) size if OneNAND case.
For this, we need to code or size optimization. If this is
solved, we can use flash as bad block free device with UBI.

An alternative to that is to have a very small IPL that loads
U-Boot to DRAM as the SPL.

Yes; our implementation of such an IPL (for PXA320) can be found
at

https://www.cvg.de/people/ensc/elito-mobm320-0.0.4.tar.bz2

This allows to jump into an image either located in a 'bootloader'
volume (which contains U-Boot) or in a 'kernel' image.

Latter is faster (by about 3-5 seconds plus the duplicate UBI
scan within U-Boot) which can make a difference as the IPL itself
needs 3.5 seconds to scan a 1GiB NAND :(

Post by Josh Boyer
That would require some relocatable support in U-Boot.

Depends on the system resp. how U-Boot was built (e.g. set its
TEXTADDR so that it does not conflict with the IPL).


Enrico