Grow UBI device?

Le Tue, 8 Oct 2013 20:28:46 +0100,

Post by Wolfgang Denk
so I can keep an existing MTD partition with a JFFS2
file system. After copying the data from JFFS2 to a UBIFS volume, I
would like to free and reuse the space of this JFFS2 partition.
Is there a way to "grow" the existing UBI device so that it now also
covers the rest of the NAND chip? Or is my only option to create a
second UBI device?

I don't remeber exactly but I think UBI (without fastmap) doesn't know
the size of the mtd partition.
The scan scan what you give to it.

A way to "grow an UBI device could be to append (before or after)
flash formated with ubi format (with --no-volume-table and the correct
--image-seq= and same other param).
If you give erased flash may be the linux driver will format it.

Matthieu

Ricard Wanderlof

2013-10-09 08:46:40 UTC

Post by Wolfgang Denk
Hi,
I'm looking for a clever way to migrate devices (in the field) from
JFFS2 to UBI/UBIFS. In the end result, I want to have single UBI
device (which covers the whole NAND chip) with several volumes used
with UBIFS. Unfortunately I must keep some of the existing data.
It would be nice if I could start with a UBI device that is smaller
than the NAND, so I can keep an existing MTD partition with a JFFS2
file system. After copying the data from JFFS2 to a UBIFS volume, I
would like to free and reuse the space of this JFFS2 partition.
Is there a way to "grow" the existing UBI device so that it now also
covers the rest of the NAND chip? Or is my only option to create a
second UBI device?

UBI devices are 'attached' to existing mtd partitions. So to grow a UBI
device, you need to detach it from the partition, re-partition the flash
(and I'm not sure if this step is even possible as it involves
unregistering existing mtd partitions), then attach a UBI device to the a
new partition.

Come to think of it, I'm not sure that would work anyway, as the
resulting, new, enlarged partition would partly contain UBI data and
partly old jffs2 data. I'm not sure what UBI does when it encounters
incorrect data, if it just erases the relevant blocks and formats them for
its own use, or if it barfs completely and just bails out complaining that
the partition does not contain UBI data. If the relevant blocks were
erased, then I think UBI would simply concede that the a previous erase
attempt had been prematurely aborted, and re-erase the blocks and write
its headers, so perhaps that is something to try. At any rate it involves
at least some mild trickery (erase the blocks that previously contained
jffs2 data).

It's probably easier to create a new UBI device by attaching it to the
partition previously used by jffs2, after erasing it, but multiple UBI
devices across a flash partly defeats the purpose of wear-leveling which
UBI provides.

/Ricard

--
Ricard Wolf Wanderl?f ricardw(at)axis.com
Axis Communications AB, Lund, Sweden www.axis.com
Phone +46 46 272 2016 Fax +46 46 13 61 30

Gupta, Pekon

2013-10-09 11:28:52 UTC

From: Ricard Wanderlof

Post by Wolfgang Denk
I'm looking for a clever way to migrate devices (in the field) from
JFFS2 to UBI/UBIFS. In the end result, I want to have single UBI
device (which covers the whole NAND chip) with several volumes used
with UBIFS. Unfortunately I must keep some of the existing data.
It would be nice if I could start with a UBI device that is smaller
than the NAND, so I can keep an existing MTD partition with a JFFS2
file system. After copying the data from JFFS2 to a UBIFS volume, I
would like to free and reuse the space of this JFFS2 partition.

That seems to be a nice move, but where would you copy the existing
JFFS2 data ?
(a) In memory, then there is threat of losing the data due to
power-failure, and you should have enough RAM.
(b) copy to target UBIFS volume/partition. So assumption is that target
UBI volume has enough space and is writable.
I'm assuming (b) is more workable ..

Post by Wolfgang Denk
Is there a way to "grow" the existing UBI device so that it now also
covers the rest of the NAND chip? Or is my only option to create a
second UBI device?

Yes, currently registering and de-registering of MTD partitions are
done as part of nand_probe() and nand_remove().
If we do on-the-fly registering / de-registering of MTD device, then we
also need to tweak the mtd partition information, which will change
- partition base offset and
- shift masks (to calculate offset within the partition)
which is ok, because they are not used by controller drivers directly.

Come to think of it, I'm not sure that would work anyway, as the
resulting, new, enlarged partition would partly contain UBI data and
partly old jffs2 data. I'm not sure what UBI does when it encounters
incorrect data, if it just erases the relevant blocks and formats them for
its own use, or if it barfs completely and just bails out complaining that
the partition does not contain UBI data. If the relevant blocks were
erased, then I think UBI would simply concede that the a previous erase
attempt had been prematurely aborted, and re-erase the blocks and write
its headers, so perhaps that is something to try. At any rate it involves
at least some mild trickery (erase the blocks that previously contained
jffs2 data).

Yes, during mounting of UBIFS volumes, UBI checks for erase-header
on first-page of every block.
- If it's able to find a valid erase-header, with correct image_seq field,
then it attaches it to its pool.
- Otherwise, if the valid erase-header was not found, then it assumes
then it ubi will try re-erasing it. And add it to its pool of free-blocks.

So, effectively you don't even need to erase the partition to convert
it into UBI volume, you just try attaching it as UBI volume and UBI
should erase it for you. (though flagging some warnings on the way).

It's probably easier to create a new UBI device by attaching it to the
partition previously used by jffs2, after erasing it, but multiple UBI
devices across a flash partly defeats the purpose of wear-leveling which
UBI provides.

True but this is good for on-field devices which are still on JFFS2 and
want to migrate to UBIFS for same reasons. Also, this would help in
other areas like recovery from power-failures.

with regards, pekon

Wolfgang Denk

2013-10-09 15:05:38 UTC

Dear "Gupta, Pekon",

Post by Ricard Wanderlof

Post by Wolfgang Denk
It would be nice if I could start with a UBI device that is smaller
than the NAND, so I can keep an existing MTD partition with a JFFS2
file system. After copying the data from JFFS2 to a UBIFS volume, I
would like to free and reuse the space of this JFFS2 partition.

That seems to be a nice move, but where would you copy the existing
JFFS2 data ?

Well, I wrote: "copy[...] the data from JFFS2 to a UBIFS volume".

Post by Gupta, Pekon
(a) In memory, then there is threat of losing the data due to
power-failure, and you should have enough RAM.
(b) copy to target UBIFS volume/partition. So assumption is that target
UBI volume has enough space and is writable.
I'm assuming (b) is more workable ..

...which is what I'm after.

Post by Ricard Wanderlof
Come to think of it, I'm not sure that would work anyway, as the
resulting, new, enlarged partition would partly contain UBI data and
partly old jffs2 data. I'm not sure what UBI does when it encounters
incorrect data, if it just erases the relevant blocks and formats them for
its own use, or if it barfs completely and just bails out complaining that
the partition does not contain UBI data. If the relevant blocks were
erased, then I think UBI would simply concede that the a previous erase
attempt had been prematurely aborted, and re-erase the blocks and write
its headers, so perhaps that is something to try. At any rate it involves
at least some mild trickery (erase the blocks that previously contained
jffs2 data).

Yes, during mounting of UBIFS volumes, UBI checks for erase-header
on first-page of every block.

Mounting UBIFS volumes should not be a problem - the existing ones are
not located in the newly added area of the NAND. The question however
is what the UBI layer itself does, if I for example try to create a
new volume which then would allocate blocks from the newly added
space.

Post by Gupta, Pekon
So, effectively you don't even need to erase the partition to convert
it into UBI volume, you just try attaching it as UBI volume and UBI
should erase it for you. (though flagging some warnings on the way).

Are you sure about that?

Well, I guess I'll have to try it out in the end...

Thanks!

Best regards,

Wolfgang Denk

Ricard Wanderlof

2013-10-09 15:45:57 UTC

Yes, during mounting of UBIFS volumes, UBI checks for erase-header
on first-page of every block.

UBI scans the whole partition when the attachement is made to the mtd
partition. So if you'd get any error message regarding blocks that lack
UBI headers it would be then, not when you subsequently try to create a
volume in the partition. By that time the volume should be "formatted"
with valid UBI headers.

Are you sure about that?
Well, I guess I'll have to try it out in the end...

I just tested it since I have a UBI system just beside me, by erasing and
then writing zeros to a block (just to put non-UBI data in it) and
restarting the system (the partition in question contains the volume with
the root file system in it, so it is attached by the kernel during its
boot). The result was that UBI didn't give any error or warning messages
during boot, but a subsequent nanddump after boot revealed that the
zeroed blocks now contained valid UBI headers, so, without looking at the
source, it seems that UBI really does manage this fairly silently.

/Ricard

--
Ricard Wolf Wanderl?f ricardw(at)axis.com
Axis Communications AB, Lund, Sweden www.axis.com
Phone +46 46 272 2016 Fax +46 46 13 61 30

Richard Weinberger

2013-10-09 18:47:42 UTC

On Wed, Oct 9, 2013 at 5:45 PM, Ricard Wanderlof

Post by Ricard Wanderlof

Yes, during mounting of UBIFS volumes, UBI checks for erase-header
on first-page of every block.

UBI scans the whole partition when the attachement is made to the mtd
partition. So if you'd get any error message regarding blocks that lack UBI
headers it would be then, not when you subsequently try to create a volume
in the partition. By that time the volume should be "formatted" with valid
UBI headers.

Are you sure about that?
Well, I guess I'll have to try it out in the end...

I just tested it since I have a UBI system just beside me, by erasing and
then writing zeros to a block (just to put non-UBI data in it) and
restarting the system (the partition in question contains the volume with
the root file system in it, so it is attached by the kernel during its
boot). The result was that UBI didn't give any error or warning messages
during boot, but a subsequent nanddump after boot revealed that the zeroed
blocks now contained valid UBI headers, so, without looking at the source,
it seems that UBI really does manage this fairly silently.

I've never tested this. But from my understanding of the UBI source
this behavior is
correct and intended. (Of course only in the non-fastmap case).

--
Thanks,
//richard

Wolfgang Denk

2013-10-09 20:01:41 UTC

Dear Ricard,

Post by Ricard Wanderlof
I just tested it since I have a UBI system just beside me, by erasing and
then writing zeros to a block (just to put non-UBI data in it) and
restarting the system (the partition in question contains the volume with
the root file system in it, so it is attached by the kernel during its
boot). The result was that UBI didn't give any error or warning messages
during boot, but a subsequent nanddump after boot revealed that the
zeroed blocks now contained valid UBI headers, so, without looking at the
source, it seems that UBI really does manage this fairly silently.

Thanks a lot, much appreciated!

Best regards,

Wolfgang Denk

--
DENX Software Engineering GmbH, MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
The IQ of the group is the lowest IQ of a member of the group divided
by the number of people in the group.

Artem Bityutskiy

2013-10-10 07:42:42 UTC

Post by Gupta, Pekon
Yes, during mounting of UBIFS volumes, UBI checks for erase-header
on first-page of every block.
- If it's able to find a valid erase-header, with correct image_seq field,
then it attaches it to its pool.
- Otherwise, if the valid erase-header was not found, then it assumes
then it ubi will try re-erasing it. And add it to its pool of free-blocks.

I think there is a separate list for blocks with garbage, and if UBI
finds too much of them, it attaches R/O. So if it would find too many
JFFS2 eraseblocks, it would not erase them. But erasing the JFFS2 blocks
from outside of UBI driver would help.

--
Best Regards,
Artem Bityutskiy

Wolfgang Denk

2013-10-09 14:33:18 UTC

Dear Ricard Wanderlof,

Post by Ricard Wanderlof

Post by Wolfgang Denk
Is there a way to "grow" the existing UBI device so that it now also
covers the rest of the NAND chip? Or is my only option to create a
second UBI device?

In my situation it is OK if the update involves a reboot, so it is
certainly possible to change the MTD partition layout. So this should
not be a problem.

well, this is the exact part where I wanted to know if (1) this is
guaranteed to work (by design), and/or (2) somebody has successfully
done it before.

Post by Ricard Wanderlof
It's probably easier to create a new UBI device by attaching it to the
partition previously used by jffs2, after erasing it, but multiple UBI
devices across a flash partly defeats the purpose of wear-leveling which
UBI provides.

Right, and it fragments the device. That's why I'd like to avoid
that.

Thanks!!

Best regards,

Wolfgang Denk

--
DENX Software Engineering GmbH, MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Doing the good deeds is like the grass in the garden. You don't see
its growth. But, it does by days. Doing the wicked deeds is like the
hone. You don't see its damage. But, it does by days. - Buddha

Artem Bityutskiy

2013-10-10 08:17:18 UTC

Wolfgang, people already answered you, and frankly I did not read all
the answers :-) But here is my brief version.

And I guess there is no temporary storage where you could copy JFFS2
files, re-format the flash, and copy the files back.

Post by Wolfgang Denk
It would be nice if I could start with a UBI device that is smaller
than the NAND, so I can keep an existing MTD partition with a JFFS2
file system.

You mean that on the existing systems, you already have an extra MTD
partition, or you somehow can create one.

Then you want to UBI/UBIFS-format the extra partition. Then copy
required JFFS2 files there. And then kill JFFS2 partitions and "merge"
them with the extra partition, and end up with UBI/UBIFS covering all
the space.

Post by Wolfgang Denk
After copying the data from JFFS2 to a UBIFS volume, I
would like to free and reuse the space of this JFFS2 partition.

OK, sounds like I understood you correctly.

Post by Wolfgang Denk
Is there a way to "grow" the existing UBI device so that it now also
covers the rest of the NAND chip? Or is my only option to create a
second UBI device?

I think so, unless you use the fastmap feature. Suppose you don't.

There are 2 parts - UBI and UBIFS.

So, let's suppose your situation is like this.

1. You have 2 mtd partitions - mtd1 and mtd2.
2. mtd1 has JFFS2 with important files.
3. mtd2 is the extra partition you have.

I guess you can do this.

1. Format mtd2 with UBI. Just erasing should be OK, or you can use
ubiformat: http://www.linux-mtd.infradead.org/faq/ubi.html#L_ubierase

Then you attach mtd2 (ubiattach) and you get an UBI device (/dev/ubiX).

2. Create a volume on /dev/ubiX which spans entire UBI device
(ubimkvol -t AFAIR). You end up with UBI volume /dev/ubiX_0.

3. And here is important part - you format /dev/ubiX_0 with JFFS2 but
correctly.

UBIFS has this "automatic expand" feature. Which means that if the UBI
volume underneath it becomes larger, UBIFS will expand automatically and
take the entire (larger) UBI volume.

However, there is a _limit_ to how much UBIFS can expand from the
original size. And you want to watch this limit and make it big enough.

The limit is connected with the "-c" option of mkfs.ubifs. Please, check
the docs - they explain what exactly -c controls and how it is related
to the maximum UBIFS file-system size:

http://www.linux-mtd.infradead.org/faq/ubifs.html#L_max_leb_cnt

So, select proper -c.

If you do not use mkfs.ubifs, but just attach /dev/ubiX_0, the UBIFS
kernel driver will format the volume automatically. It will select the
-c value itself. Since this value is power of 2, it will just take the
smallest power-of-2 value. It may be OK for your purposes, may be not.
It is safer to use mkfs.ubifs with explicit -c.

Basically this is the trickiest part, rest _should_ be easy, but you'll
learn when you try :-)

4. Copy important files from JFFS2-mtd1 to UBIFS-UBI-mtd2.

5. Erase mtd1 (or you can do this later too)

6. I do not know how you do this, but kill mtd1, and resize mtd2.
Probably mtd2 will then be named mtd1 after reboot. If you always refer
your MTD partitions and UBI volumes by name, you should not suffer to
much from mtd2 becoming mtd1.

Lets suppose there is no rename, for simplicity.

7. Make sure former JFFS2-mtd1 eraseblocks are erased (unless you did
this at step 5).

8. Attach mtd2 to UBI. UBI will automatically "take" the new
eraseblocks, erase them and write UBI headers there, and make available.

9. Volume size will still be the same, and you need to re-size it using
ubiresize.

10. Mount the resized volume. UBI will enlarge itself.

Problem solved.

Surely you do not want power cuts during this operations. If you need to
survive power cuts, I guess it is possible, but more complex.

HTH.

--
Best Regards,
Artem Bityutskiy

Richard Weinberger

2013-10-10 08:34:44 UTC

Post by Artem Bityutskiy
Wolfgang, people already answered you, and frankly I did not read all
the answers :-) But here is my brief version.

And I guess there is no temporary storage where you could copy JFFS2
files, re-format the flash, and copy the files back.

Post by Wolfgang Denk
It would be nice if I could start with a UBI device that is smaller
than the NAND, so I can keep an existing MTD partition with a JFFS2
file system.

You mean that on the existing systems, you already have an extra MTD
partition, or you somehow can create one.
Then you want to UBI/UBIFS-format the extra partition. Then copy
required JFFS2 files there. And then kill JFFS2 partitions and "merge"
them with the extra partition, and end up with UBI/UBIFS covering all
the space.

Post by Wolfgang Denk
After copying the data from JFFS2 to a UBIFS volume, I
would like to free and reuse the space of this JFFS2 partition.

OK, sounds like I understood you correctly.

Post by Wolfgang Denk
Is there a way to "grow" the existing UBI device so that it now also
covers the rest of the NAND chip? Or is my only option to create a
second UBI device?

s/JFFS2/UBIFS :-)

Post by Artem Bityutskiy
UBIFS has this "automatic expand" feature. Which means that if the UBI
volume underneath it becomes larger, UBIFS will expand automatically and
take the entire (larger) UBI volume.
However, there is a _limit_ to how much UBIFS can expand from the
original size. And you want to watch this limit and make it big enough.
The limit is connected with the "-c" option of mkfs.ubifs. Please, check
the docs - they explain what exactly -c controls and how it is related
http://www.linux-mtd.infradead.org/faq/ubifs.html#L_max_leb_cnt
So, select proper -c.
If you do not use mkfs.ubifs, but just attach /dev/ubiX_0, the UBIFS
kernel driver will format the volume automatically. It will select the
-c value itself. Since this value is power of 2, it will just take the
smallest power-of-2 value. It may be OK for your purposes, may be not.
It is safer to use mkfs.ubifs with explicit -c.
Basically this is the trickiest part, rest _should_ be easy, but you'll
learn when you try :-)
4. Copy important files from JFFS2-mtd1 to UBIFS-UBI-mtd2.
5. Erase mtd1 (or you can do this later too)
6. I do not know how you do this, but kill mtd1, and resize mtd2.
Probably mtd2 will then be named mtd1 after reboot. If you always refer
your MTD partitions and UBI volumes by name, you should not suffer to
much from mtd2 becoming mtd1.
Lets suppose there is no rename, for simplicity.
7. Make sure former JFFS2-mtd1 eraseblocks are erased (unless you did
this at step 5).
8. Attach mtd2 to UBI. UBI will automatically "take" the new
eraseblocks, erase them and write UBI headers there, and make available.
9. Volume size will still be the same, and you need to re-size it using
ubiresize.
10. Mount the resized volume. UBI will enlarge itself.
Problem solved.
Surely you do not want power cuts during this operations. If you need to
survive power cuts, I guess it is possible, but more complex.
HTH.
--
Best Regards,
Artem Bityutskiy
______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

--
Thanks,
//richard

Wolfgang Denk

2013-10-15 17:40:31 UTC

Dear Artem,

Post by Artem Bityutskiy
Wolfgang, people already answered you, and frankly I did not read all
the answers :-) But here is my brief version.

Thanks.

And I guess there is no temporary storage where you could copy JFFS2
files, re-format the flash, and copy the files back.

Correct.

Post by Wolfgang Denk
It would be nice if I could start with a UBI device that is smaller
than the NAND, so I can keep an existing MTD partition with a JFFS2
file system.

You mean that on the existing systems, you already have an extra MTD
partition, or you somehow can create one.
Then you want to UBI/UBIFS-format the extra partition. Then copy
required JFFS2 files there. And then kill JFFS2 partitions and "merge"
them with the extra partition, and end up with UBI/UBIFS covering all
the space.

Correct.

Post by Wolfgang Denk
Is there a way to "grow" the existing UBI device so that it now also
covers the rest of the NAND chip? Or is my only option to create a
second UBI device?

I think so, unless you use the fastmap feature. Suppose you don't.

We don't. We may want to do this later, but certainly not during this
migration steps.

Post by Artem Bityutskiy
1. Format mtd2 with UBI. Just erasing should be OK, or you can use
ubiformat: http://www.linux-mtd.infradead.org/faq/ubi.html#L_ubierase
Then you attach mtd2 (ubiattach) and you get an UBI device (/dev/ubiX).
2. Create a volume on /dev/ubiX which spans entire UBI device
(ubimkvol -t AFAIR). You end up with UBI volume /dev/ubiX_0.
3. And here is important part - you format /dev/ubiX_0 with JFFS2 but
correctly.

With JFFS2? I would expect to see "UBIFS" here?

Thanks for the poointer. I would have missed this.

Post by Artem Bityutskiy
If you do not use mkfs.ubifs, but just attach /dev/ubiX_0, the UBIFS
kernel driver will format the volume automatically. It will select the
-c value itself. Since this value is power of 2, it will just take the
smallest power-of-2 value. It may be OK for your purposes, may be not.
It is safer to use mkfs.ubifs with explicit -c.

Agreed. It would have worked in our case, as mtd1 is 32 MB and mtd2
is 96 MB on a 128 MB NAND device, but it's much better to have full
control.

Post by Artem Bityutskiy
4. Copy important files from JFFS2-mtd1 to UBIFS-UBI-mtd2.
5. Erase mtd1 (or you can do this later too)
6. I do not know how you do this, but kill mtd1, and resize mtd2.
Probably mtd2 will then be named mtd1 after reboot. If you always refer
your MTD partitions and UBI volumes by name, you should not suffer to
much from mtd2 becoming mtd1.

The plan is to just have another mtd3 which covers the whole device.
So I would detach UBI from mtd2 and re-attach to mtd3.

Post by Artem Bityutskiy
Lets suppose there is no rename, for simplicity.
7. Make sure former JFFS2-mtd1 eraseblocks are erased (unless you did
this at step 5).
8. Attach mtd2 to UBI. UBI will automatically "take" the new
eraseblocks, erase them and write UBI headers there, and make available.
9. Volume size will still be the same, and you need to re-size it using
ubiresize.
10. Mount the resized volume. UBI will enlarge itself.

That's just amazing :-)

Post by Artem Bityutskiy
Problem solved.
Surely you do not want power cuts during this operations. If you need to
survive power cuts, I guess it is possible, but more complex.

Thanks a lot for all the help, it's highly appreciated!

Best regards,

Wolfgang Denk

Richard Weinberger

2013-10-15 18:38:47 UTC

Post by Wolfgang Denk
Dear Artem,

Post by Artem Bityutskiy
Wolfgang, people already answered you, and frankly I did not read all
the answers :-) But here is my brief version.

Thanks.

And I guess there is no temporary storage where you could copy JFFS2
files, re-format the flash, and copy the files back.

Correct.

Post by Wolfgang Denk
It would be nice if I could start with a UBI device that is smaller
than the NAND, so I can keep an existing MTD partition with a JFFS2
file system.

You mean that on the existing systems, you already have an extra MTD
partition, or you somehow can create one.
Then you want to UBI/UBIFS-format the extra partition. Then copy
required JFFS2 files there. And then kill JFFS2 partitions and "merge"
them with the extra partition, and end up with UBI/UBIFS covering all
the space.

Correct.

Post by Wolfgang Denk
Is there a way to "grow" the existing UBI device so that it now also
covers the rest of the NAND chip? Or is my only option to create a
second UBI device?

I think so, unless you use the fastmap feature. Suppose you don't.

We don't. We may want to do this later, but certainly not during this
migration steps.

With JFFS2? I would expect to see "UBIFS" here?

Yes, UBIFS.