This post is about a few interesting bytes in pe2usb.bin, a boot sector image. The source of the file is Bart's PE builder v3.1.10a, and is used by the pe2usb tool as part of creating a bootable Bart PE USB drive. I am trying to understand the significance of these bytes.

I understand that there are now more modern and advanced techniques for making a USB stick bootable, so I may be resurrecting an old topic here. It is mostly for the sake of intellectual curiousity, but I hope to gain understanding from it. I have not been able to find the answer to the following question on these forums so far.

pe2usb.bin appears to be a 512-byte boot sector image for a FAT16 partition, loading NTLDR. I have compared it against the standard FAT16 boot sector created by any Windows NT 5.x version (XP for example), or by Windows 7's bootsect /nt52 command. It mostly matches up. However, ignoring the differences in the BPB (which are necessarily drive-specific), I have found the following differences in the boot loader code: (All values are in hex)
I only have a limited knowledge of x86 assembly, so I would like help with understanding the purpose of these alterations. I found the following thread: Boot BartPE from USB Flash Drive, which shows that Bart made these changes on purpose, whilst he was developing the pe2usb method. Sadly I can't find him explaining their role. (Funnily enough, in the opening post he accidentally had the images showing the change backwards, but noone in the thread seemed to have picked up on it). That thread actually only mentions the first two changes, the last change at 17C seemed to have been introduced later.

Unfortunately the FAT16 boot sector code under NT 5.x is not analyzed by the Starman at http://thestarman.narod.ru/asm/mbr/index.html and this code is different from FAT32 or from FAT16 under earlier or later OS's.

From my limited knowledge of x86 assembly,
so a comparison/conditional jump is being replaced with an unconditional jump. I haven't made progress on the other differences. I don't even understand why replace the NOP at 03 (which is jumped over anyway).

Background motivation:
I have an old-ish Dell Inspiron 6400 laptop and was playing around with USB booting on it with a 1Gb Kingston DataTraveler stick. I ran into a caveat when I formatted the primary partition of the USB stick to FAT16 CHS (partition type 06), and using standard NT 5.x boot sector code. During booting, the stick will only load NTLDR if it was one of the first files copied to it. If another large file (for example PE iso image) was copied to the stick first, the laptop just hangs. I found two workarounds, both worked perfectly: Either change the partition to FAT16 LBA (type 0E), or use the boot sector code from pe2usb.bin.

What this shows is that on this laptop there is some issue (probably in BIOS) with handling CHS correctly, which is making deeper sectors on the stick inaccessible during boot. The code in pe2usb.bin seems to work around it, and I would love to know what it does differently.

PS: This is my first post on these forums, so I want to say thank you for the amazing resources available on here. I have just recently come across this place and I am learning a lot.

NOT EXACTLY what you asked for , but do take some time here:
http://www.911cd.net/forums//index.php?sho...1702&st=129
and here (FAQ #10):
http://jaclaz.altervista.org/Projects/USB/USBfaqs.html

And here:
http://www.911cd.net/forums//index.php?showtopic=23168

You may get some hints in the right direction.

Check also Ray Knight's pages:
http://www.rayknights.org/



jaclaz

The first bit of code at 50h replaces a



with



so this is very sensible for a USB drive. If the USB drive booted as 80h (HDD0) it overwrites the BPB with the 80h. RMPrepUSB always puts 80h here anyway if you format as a HDD.

the 17C patch was

but NOPing it out means the code will always go to 17E

looking at the code just before 17C, the patch is to always force the code to use Int 13h Extended calls (AH=42) rather than Standard (AH=02).
Interestingly this code is


so it seems that if byte 02 is E (I assume you mean offset 2 - the third byte, not offset 3???) it will use use Extended int 13h BIOS calls. This is why the instruction at the start must be a 2 byte jmp and not a 3 byte jump. The BIOS or OS must put 0E here if it wants to use extended LBA mode. This explains why your BIOS needs this patch as your boot files must be beyond the reach of standard CHS Int 13 calls!
So this must be what the patch at byte 02 was for, but actually it probably is not used in the code anyway as this patch at 017C has patched it out anyway!

P.S. Looking at the code, it seems that area (start of memory where PBR is loaded) is used as a scratchpad by the code (e.g. calculates LBA, etc.) so it may be that the author thought he could force LBA by using E at offset 2 and then when it did not work, patched the code instead but still left the byte at offset 02 set to 0E?????


Interesting!

Thank you both for the very speedy response! It seems clear to me that this is a CHS addressing issue.

@jaclaz: You are a veritable trove of knowledge I have read your FAQ already, but your (indirect) link to Removing CHS based access from windows boot loaders was interesting, and gave me an idea.

In order to speed things up when making a change, I was preparing (format etc.) my USB stick on my desktop PC (with a modern P8H67-M Asus motherboard) and trying it out in the Dell laptop. The two BIOS-es might be reporting different CHS geometry for this stick, I need to look into this further.

I have come across Ray Knight's pages as well, unfortunately like the Starman, he doesn't analyze an NT 5.x FAT16 boot sector code either. I have done a fair bit of Googling and I haven't come across a suitable page yet. All the boot sector code analysis always seems to focus on the latest filesystem version in each OS version (FAT32 in Windows 98/ME and NTFS in Windows 2k/XP and above).

@ Steve6375: That is amazing, thank you very much for that analysis! You are right, I meant offset 02. Do you happen to have a complete analysis of NT 5.x FAT16 boot sector code somewhere, or did you just do it on the spot? I would love to read a full analysis of this particular boot sector code. A lot of the advice I read about USB booting suggests to use FAT16 for greatest compatibility, and Windows XP is used very often when working with USB booting.


Based on this, the patch at offset 50h is a workaround for issues with the drive number. I did take care to make sure my stick has 80h in the BPB. As far as I know however, when Windows XP formats a USB stick partition, it puts 00h in there, so maybe this was a fix for that case. Assuming the BIOS correctly reports 80h. At any rate, this doesn't seem to be what's doing the trick for me.

This must be what's doing the trick for me then.

Chronologically, this must have been the case.

This leads me to a new question: if the original unpatched code is "cmp [byte ptr BP+2],0E", the value 0E that is compared against must come from an external source. You mention BIOS or OS. It can't be the OS, as there is no OS running yet (this is at boot time). Is it the BIOS? Or is it the partition table entry? Is this 0E the same as the partition type 0E for FAT16 with LBA?

Hi, re. byte at offset 2. I meant that whatever wrote the original PBR maybe writes E at this location, however I have not fully analysed the PBR code and so it may be that the PBR code uses this location as a scratchpad byte, determines if LBA is supported and if it is then changes this location to E. I just don't know.

re. looking at the code. What I did was to prepare a USB stick using RMPrepUSB and the FAT16+NTLDR+Boot as HDD option. Then install grub4dos onto the MBR and then put FreeDOS files on it (included in RMPrepUSB download) and symdeb.exe (or you could use debug.exe or any DOS debugger) on the stick. Then I booted from it using Boot USB as C: option from the menu and used the command:

L 2000:1000 2 0 1 (load first sector of volume C:) 2=C: 0=sector address 1=num of sectors to read
U 2000:1000 (unassemble at this address)

The only problem with debug/symdeb is that is does not understand 32-bit code (66h pre-cursor byte for 32-bit instructions) and some other new opcodes.

P.S. the current versions of RMPrepUSB use the standard XP PBR for FAT16 NTLDR PBR. I have added the patches you found to a new version 2.1.623 (not yet on website) for both FAT16+NTLDR and FAT16+Bootmgr PBR versions as they both use the same code. If you want it now I can upload a Beta 2.1.623 for you to try?

Very good.

Another patch that may be needed is however "before" bootsector comes into play, into the actual MBR.

(of course this applies to the 2K/XP MBR and not to grub4dos)

It is something I found by pure chance, a patch the HP USB tool makes:
http://reboot.pro/2246/page__st__15
which I added to MBRBatch:
http://reboot.pro/3191/
and that later was used in fuwi's utility too.


jaclaz

RMPrepUSB use the Vista/7 MBR code. I think the reason for the patch in the XP MBR is that the preceding code requests the Disk Parameters from the BIOS and then multiplies out the max cyl/cyls per hd/spt and compares it with the LBA address in the active partition table - if the LBA address is a lower number than the max sector LBA address of the hard disk then it uses standard int 13h calls but it is always safest to use Extended Int13h calls for USB devices as they have no Cyl/Hd/Sec geometry, so if standard INT 13 calls were used the BIOS would need to translate CHS into LBA and may not do it correctly (especially if the BIOS or the program that wrote the partition table parameters assumes/assumed 240 hds per cyl!).
Vista/Win7 MBR code always uses Extended Int 13h unless the BIOS reports it is not available.

@jaclaz: I have found before your analysis of the byte differences introduced in the MBR by the HP USB Tool. It was a very interesting read, and it partially prompted my desire to understand the byte differences in pe2usb.bin

@Steve: I haven't used RMPrepUSB before, but I have come across it. Does PBR stand for "Primary Boot Record", and does it refer to the boot sector at the start of the partition (as opposed to the MBR)? Thank you for your explanation on using debug.

Personally, I prefer not to have small few byte patches in the boot code. I prefer to work with the standard code as much as possible, as that's what you can expect to be created by standard tools (such as formatting under XP). I was studying pe2usb.bin mostly for intellectual curiousity, and to understand limitations of the standard code However, if you would like me to try out the Beta verison of RMPrepUSB for you, I'd be happy to report back on it.

Based on Steve's analysis, I thought we now understand what's going on:
* The 4-byte patch @ 50h is to safeguard against incorrect drive number
* The 2-byte patch @ 17Ch is to ensure LBA access.
So I went about verifying this. Armed with the trusty Tiny Hexer, I did a series of tests (see below). The results were mostly what you'd expect, but I found one interesting thing: the 1-byte patch at @ 02h does have a significance.

Preparation:
Format the USB to hard disk type, with MBR and a single FAT16 primary partition. Use standard (unpatched) NT 5.x MBR and PBR codes, such as found in Windows XP. The focus of the testing was the PBR code, so I didn't touch the MBR code. Set the partition active. I copied a random 165 Mb file on to the USB, followed by ntldr and ntdetect.com. Those two are enough to verify that the USB is booting. The USB was tested in the Dell Inspiron 6400 laptop.

Variables to play with:
* Partition type: 06h and 0Eh
* Drive number in BPB: 00h and 80h
* 1-byte patch @ 02h
* 4-byte patch @ 50h
* 2-byte patch @ 17Ch

Baseline observations without patch:
* Drive number 00h, any partition type: "Disk Error" message
* Drive number 80h, partition type 06h: Hangs
* Drive number 80h, partition type 0Eh: Boots

I did a series of tests on the effect of the patches, here is a summary of my observations:
* Drive number 80h, any partition type, only patch @ 17Ch: Boots
* Drive number 00h, any partition type, only patch @ 17Ch: "Disk Error"
* Drive number 00h, any partition type, only patch @ 50h: Hangs
* Drive number 00h, any partition type, patch @ 02h and 50h: Boots
* Drive number 00h, any partition type, patch @ 50h and 17Ch: Boots
* Drive number 00h, any partition type, all 3 patches: Boots
The unexpected result here was that with only patch @ 50h, the USB didn't boot.

Conclusions:
* The patch @ 17Ch can be used on its own to force LBA access.
* The patch @ 02h and 50h together can be used to bypass drive number check, and apparently force LBA access as well in my case.
* The patch @ 50h can't be used on its own.

The sequence of events in the boot code must be (among other things):
1) Check drive number
2) Query the partition type and store it at 02h
3) Later on, check partition type at 02h for LBA access
Applying the patch @ 50h must be bypassing both 1 and 2, hence the need to patch @ 02h as well. Applying the patch @ 17Ch is bypassing the check of 02h, so doesn't matter if that is patched or not.

Interesting.

Again, very good.

BUT (isn't there alway a BUT?) did you carry your experiments with a "balanced" CHS/LBA (such as a normally working partitioning utility would make) or with an UNbalanced CHS/LBA such as the one the HP utility often makes?

And HOW big in size was the actual USB stick?

It is very possible that with a stick smaller than the first CHS limit (the around 528 Mb one) , and depending on the actual BIOS code, the behaviour changes when compared to a bigger stick.

And additionally it is possible that using a "fake second partition" (corresponding to the 2PTN setting in RMPREPUSB), things change again.

For the record, PBR (Partition Boot Record) is "another name" (actually slightly - but not entirely - more accurate) for the "bootsector" (that on NTFS for example is more than one sector long).
The most "exact" name is VBR (Volume Boot Record), since also Logical Volumes inside Extended have it (and can be made bootable).


jaclaz

Let me answer these thoroughly (fortunately I have written down the data ). I am unsure what exactly you are referring to by "balanced" CHS/LBA, so I will just give you the data.

The USB stick physically has 0x1DD000 sectors of size 0x200, so a total size of 0x3BA00000 (1,000,341,504 bytes), just slightly under 1 Gb.

When partitioning it, different tools choose a slightly different sector as the first one on the sitck. The partition table entry with different tools looks like this:

where "Original" is what the stick came with, "W7 diskpart + GUI" means partition with diskpart and format via Windows Explorer, and "W7 Disk Management" is the Windows 7 tool of that name.

The Num LBA is such as to fill up the whole USB stick in every case except partitioning via Disk Management. The Num LBA doesn't match up with the End CHS values in any of these cases. A total CHS sector count of (121, 255, 63) is only 0x1DA939 in LBA. Is this what you mean by unbalanced?

When formatting to FAT16, the relevant entries in the BPP in the VBR are:
Small sectors : 0
Sectors per track: 0x3F
Number of heads: 0xFF
Hidden sectors: 0x3F (varies)
Large sectors: 0x1DCFC1 (varies)
Naturally, Hidden sectors and Large sectors varies depending on the partition size.

If I understand it correctly, what matters for the VBR boot code is the "Sectors per track" and "Number of heads" recorded in the BPB. Does the VBR code make use of any CHS values from the PT entry at all? I think it's not possible to format a stick in a "BIOS dependent" way, so does the info in the BPB only depend on the formatting tool?



Is this first CHS limit the one established by the 10:4:6 bit restriction on CHS, giving a maximum of 1024*16*63 sectors? This could make a difference. However, reading here:
http://en.wikipedia.org/wiki/Logical_block_addressing
this limit came about due to a clash with the ATAPI standard, which is only relevant for IDE devices, so I am not sure it's applicable to a USB stick. Sadly I don't have any USB sticks below 1 Gb to try with.

I suppose I could try to experiment with manually hexediting the PT and BPB entries to use only 16 heads (upping the cyl count appropriately), and see what that does.

PS: Thanks for participating in this discussion with me, since it serves very little purpose beyond intellectual stimulus

Yes.

If you have "balanced data" (meaning that NO matter if you use CHS data or LBA data you always have the same start and end sector) you won't need most (if not all) of these patches, AND some BIOSes may be "sensible" to "orthodox" partitioning, i.e. having both beginning and end sector on Cylinder and Head boundary.

As you can see on the (everyday more outdated page):
http://jaclaz.altervista.org/Projects/USB/USBstick.html
the actual "original" by having the start of the partition on the SAME sector no matter which geometry the BIOS assigns to the device, allows to "reach it" on *any* BIOS.

Compare also with the actual "triple way" partitioning/formatting, see here:
http://reboot.pro/7507/


Yes and no.
The actual scope of the bootsector patches (at least the ones for FAT32 and NTFS I linked you to - but it seems like the one you found are the corresponidng one for FAT16) only LBA is used (and CHS is simply "ignored").

The general idea for which I wrote the MBRbatch was actually that of making sure that CHS and LBA are always "balanced" as to prevent possible errors.
The idea is/was that if you create a partition table as "standard" as possible it will have the more chances of working on the most hardware.

And anyway you won't ever be able to cover "everything" BIOSes are "strange beasts", and manufacturers have a quirk for introducing "strange checks" or "queer code", a few examples of such:
http://reboot.pro/13205/
http://reboot.pro/10503/
http://reboot.pro/12436/
http://reboot.pro/4863/
and of course the mentioned "2nd fake partition" that has proved in some cases to be able to switch on some BIOSes between "floppy-like" and "hd-like":
http://reboot.pro/7704/

Yet another topic revolving around the possible issues:
http://reboot.pro/13675/


It's much more than that, it is "life":




jaclaz

Thanks for all the links jaclaz, they were interesting.

Success! I managed to boot from FAT16 CHS (type 06) with files arranged as before. No hanging Here is what I did:

I reflected on all the posts before, links I read and the data I had. I concluded that:
* This is not an issue with the BIOS correctly accessing the drive.
* This is not a drive-type issue. The stick is being recognized/booted as HDD with drive number 80h. No need for 2nd fake partition etc.
* This is not an issue with the location of the starting sector of the partition. A number of partition start CHS configurations worked (0,0,33; 0,1,1; 0,2,3). The VBR is getting loaded successfully under every scenario. No need for "triple way" partitioning.
* This is not an issue with accessing absolute sectors. LBA access works perfectly.
* This IS an issue with CHS geometry when accessing a file semi-deep on the drive.
* The issue arises when the VBR code tries to use CHS.

Where I could go from here:
1) Check if this is a CHS/LBA balancing issue, or issue with the 528 Mb limit.
2) Check if this is an issue with the way the BIOS is emulating the CHS geometry of the drive.

Checking for CHS/LBA balance and 528 Mb limit

I doubted that this is the problem, because the only place where this data appears is in the PT entry, and the VBR code is not making use of the CHS data. However, I checked it out. Without going into all the details:
* A partition ending on a CHS count of 1023, 15, 63, and with Number of heads = 16 in the BPB didn't work out, same issue as before. I made sure the CHS/LBA values in the PT are balanced.
* A 500 Mb partition with balanced CHS/LBA values, but using 255 heads, didn't work out.
* Couple other small tests I could think of, didn't work out.

Checking for issue with the way BIOS emulates the CHS geometry

At this point, based on everything before, my suspicion was that the BIOS is presenting/using an incorrect geometry to emulate CHS on the stick. The issue wouldn't be about size, but rather shape. The way to test this is to see what CHS geometry the BIOS thinks the stick has. This can be done via INT 13 function 8 (GetDriveParameter).

Thanks to jaclaz pointing me at cod11's post here: http://reboot.pro/12436/ which had the perfect tool. I loaded cod11's the MBR-SPY.REC code onto my stick's MBR, made a couple small changes, and did my tests. This tool calls INT 13 function 8 immediately after loading the MBR and displays the result.

The results: (3-bytes HSC is in the same order as in PT entry)

where Laptop is Dell Inspiron 6400, and Desktop has an Asus P8H67-M motherboard. The Desktop had USB mode set to Auto in the BIOS. The difference between Desktop and Desk2 is that for Desktop, the USB stick had a nulled out PT (and no VBR), whereas for Desk2 it had a properly formatted PT and VBR. More on this below.

So what I saw is that the Dell Laptop was emulating the drive to have 32 heads; this is where the problem was all along! All the tools I used seemed to have all partitioned/formatted the stick assuming 255 heads. From here, finding a solution was simple. I prepared the USB stick again from scratch via standard tools and formatted it to FAT16 CHS (type 06). I then hexedited the End CHS value to the 3-bytes 1F FF C8 (max CHS 968, 31, 63), and the BPB Number of heads to 32. The BPB Sectors per track was left at 63. I copied the files on the stick as before, and this time it successfully booted!

The significance of the Asus result

The Asus BIOS seems to be very smart about what geometry it emulates, and is going out of its way to try and be compatible. It is actually reading the BPB's Sectors per track and Number of heads information from the VBR, and emulates the CHS geometry to match that! Only if it can't read this info, then it falls back to a default geometry. I explicitly tested this by entering nonstandard numbers in the BPB, and sure enough the Asus BIOS emulated a CHS geometry accordingly.

This post concludes my foray into this area, I am now happy that I managed to boot a FAT16 CHS stick on my Dell laptop. It's been a fun ride

Very nice work!


jaclaz

P.S.: This is only seemingly unrelated, it may be of use (32 bit systems only):
http://www.forensicfocus.com/index.php?nam...opic&t=4805

I am attaching BOTH the "compact" 1,024 bytes version and the (possibly) understandable "full" one.

Nice tool! I particularly like the effort you put into the compact one