Simple FAT and SD Tutorial Part 1

Are you limited by 128 bytes of EEPROM on your MCU or even the few kilobytes of flash in your project? Instead of just downloading a library like Petit FAT File System Module and following blindly a tutorial on how to customize it to your microcontroller and SD card, would you like to really understand what you are doing, and maybe learn a bit about filesystems and SPI in the process?

In this first part of my FAT and SD tutorial, we’ll take a SD card image, and create a simple C program to interpret its contents. For this part, you don’t need any hardware at all, just a computer with gcc (GNU C Compiler) or any other ANSI C compatible compiler installed.

Getting ready: Hex editor and disk image

To make the coding easier, I recommend a good hex editor. The one I’m using is the free and excellent HxD by Maël Hörz. You can also use it to create a 1:1 disk image from a physical SD card. To have a filesystem to read, I purchased a 1 GB micro-SD card with SD adapter for 5€, plugged it into my computer and formatted it as FAT16 (over 2 GB cards will likely get formatted as FAT32), and copied Hamlet from Project Gutenberg and some other dummy test files to it (also created a subdirectory with a few text files in it):



I then launched HxD in Administrator mode to allow it to open physical disks. Note that you don’t open just the one FAT16 partition (“H:”), but the whole removable disk (I circled the icon you need to click first):

Because the partition data, file system and few test files only occupy only the very beginning the 1 GB disk, I then proceeded to select only the first 1 MB (Edit > Select block… > Offset 0-FFFFF) and copy-pasted that into a new file, which I then saved as test.img in my project folder. If you don’t have a smallish SD/micro-SD card at hand, you can just grab the project zip and use my test image.

Note: If you’re using Linux or a Mac, you can just use dd if=/dev/sda of=~/test.img bs=1M count=1 to create the image (assuming the kernel selected /dev/sda as the device for your card). You may need to add “sudo” in the beginning.

Reading the partition data

Now that we have a file image of the SD card (or the beginning of it), we can start poking around. To get going, I first read the FAT on [SD cards] tutorial by Thiadmer Riemersma and Claudio Toffoli of CompuPhase, and I really suggest you to read the 1 page of text under the heading “The Master Boot Record and the partition table” at this point. In short, we expect the test.img to be roughly laid out like this:

  • MBR and partition table
  • …maybe some sectors for logical partitions or reserved space…
  • Boot sector of a FAT16 partition
  • …optionally some reserved sectors…
  • 1-2 copies of File Allocation Table (FAT)
  • Root directory
  • Other directories and data

Update 2013-08-20:: As one of my readers pointed out, some SD/micro-SD cards don’t have several partitions and MBR at all, but are formatted like floppy disks of old. These types of media start straight with a boot sector, described in next section.

Before we go further, you should know that the basic storage block in filesystems is called a sector, and that is simply a 512-byte (256-word) block of data. The MBR and partition table take up one sector. Boot sector will take up one sector. The copies of FAT take some N sectors, and so on. Diskettes and older operating systems used a so-called CHS (cylinder, head, sector) addressing where the sector was the smallest unit, but we don’t need to know about cylinders and heads, just the sector will do for now.

From the tutorial I linked above, we learn that the partition table should start at offset 0x1BE (hex numbers are used a lot) and contain four 16-byte partition entries, each laid out in the following manner:

Offset Description Size
0 0×80 if active (bootable), 0 otherwise 1
1 Start of the partition in CHS-addressing 3
4 Type of the partition 1
5 End of the partition in CHS-addressing 3
8 Relative offset to the partition in sectors (LBA) 4
12 Size of the partition in sectors 4

The partition type should be 4 for FAT16 partitions that are less than 32 MiB (MiB = 1024^2 bytes), 6 for over 32 MiB partitions and 14 for FAT16 partitions using LBA addressing. Let’s try reading the partition table in C:

#include <stdio.h>
#include <stdlib.h>

int main() {
    FILE * in = fopen("test.img", "rb");
    unsigned int i, start_sector, length_sectors;
    
    fseek(in, 0x1BE, SEEK_SET); // go to partition table start
    
    for(i = 0; i < 4; i++) { // read all four entries
        printf("Partition entry %d: First byte %02Xn", i, fgetc(in));
        printf("  Partition start in CHS: %02X:%02X:%02Xn", fgetc(in), fgetc(in), fgetc(in));
        printf("  Partition type %02Xn", fgetc(in));
        printf("  Partition end in CHS: %02X:%02X:%02Xn", fgetc(in), fgetc(in), fgetc(in));
        
        fread(&start_sector, 4, 1, in);
        fread(&length_sectors, 4, 1, in);
        printf("  Relative LBA address %08X, %d sectors longn", start_sector, length_sectors);
    }
    
    fclose(in);
    return 0;
}

Save this as read_mbr.c in the same folder that you saved test.img, and compile with gcc read_mbr.c -o read_mbr.exe and run:

Pretty easy, huh! But reading every field manually gets cumbersome after a while, so we could just define everything in a nice struct. Here’s the modified read_mbr2.c:

#include <stdio.h>
#include <stdlib.h>

typedef struct {
    unsigned char first_byte;
    unsigned char start_chs[3];
    unsigned char partition_type;
    unsigned char end_chs[3];
    unsigned long start_sector;
    unsigned long length_sectors;
} __attribute((packed)) PartitionTable;

int main() {
    FILE * in = fopen("test.img", "rb");
    int i;
    PartitionTable pt[4];
    
    fseek(in, 0x1BE, SEEK_SET); // go to partition table start
    fread(pt, sizeof(PartitionTable), 4, in); // read all four entries
    
    for(i=0; i<4; i++) {
        printf("Partition %d, type %02Xn", i, pt[i].partition_type);
        printf("  Start sector %08X, %d sectors longn", 
                pt[i].start_sector, pt[i].length_sectors);
    }
    
    fclose(in);
    return 0;
}

Note that I printed out only the minimum needed information this time. The __attribute((packed)) is needed so the compiler won’t align our data structure fields to 32-bit or 64-bit boundaries (by default, it likes to pad structures with empty areas so the processor can handle the data a bit faster) and make a mess. Also, I’m using a compiler where sizeof(char) is 1, sizeof(short) is 2 and sizeof(long) is 4, adjust your data types if you’re using a different system or compiler!

Reading and interpreting the boot sector

Based on the previous output, it is rather evident that the first partition is the one we want to examine further. I now advise you to read the chapter “FAT and the Boot Sector” from CompuPhase tutorial. Based on the tutorial, the boot sector for a FAT16 file system should nicely fit into this structure:

typedef struct {
    unsigned char jmp[3];
    char oem[8];
    unsigned short sector_size;
    unsigned char sectors_per_cluster;
    unsigned short reserved_sectors;
    unsigned char number_of_fats;
    unsigned short root_dir_entries;
    unsigned short total_sectors_short; // if zero, later field is used
    unsigned char media_descriptor;
    unsigned short fat_size_sectors;
    unsigned short sectors_per_track;
    unsigned short number_of_heads;
    unsigned long hidden_sectors;
    unsigned long total_sectors_long;
    
    unsigned char drive_number;
    unsigned char current_head;
    unsigned char boot_signature;
    unsigned long volume_id;
    char volume_label[11];
    char fs_type[8];
    char boot_code[448];
    unsigned short boot_sector_signature;
} __attribute((packed)) Fat16BootSector;

To make the code a bit more robust, I used a for loop that stops when it finds the first partition with compatible partition type (4, 6 or 14), so the partition entry is in pt[i]. To move inside the image file, we’ll first use the fseek() command, then read the boot sector:

fseek(in, 512 * pt[i].start_sector, SEEK_SET);
fread(&bs, sizeof(Fat16BootSector), 1, in);

I added a few printout commands and saved the result into read_boot.c – you should understand it easily when comparing with read_mbr2.c. The results of a test run can be seen on the right.

Reading the root directory

Before we dive deep into the file allocation table, we’ll peek ahead and read the root directory. To do that, we need to skip over any reserved sectors (amount of reserved sectors is bs.reserved_sectors, which includes the boot sector we just read) and all FATs (there are bs.number_of_fats of them, each bs.fat_size_sectors sectors long):

fseek(in, (bs.reserved_sectors-1 + bs.fat_size_sectors * bs.number_of_fats) *
      bs.sector_size, SEEK_CUR);

Note that we’re using the SEEK_CUR seek mode, which uses the current location as a base. This statement is added after reading the boot sector, which is why we compensate for it by substracting one from bs.reserved_sectors.

For this part, I’m using the Phobos FAT file system tutorial as a reference. It’s more detailed on the FAT part than the previous tutorial. “The Root Directory” chapter contains the information we need to decipher the structure of file entries:

typedef struct {
    unsigned char filename[8];
    unsigned char ext[3];
    unsigned char attributes;
    unsigned char reserved[10];
    unsigned short modify_time;
    unsigned short modify_date;
    unsigned short starting_cluster;
    unsigned long file_size;
} __attribute((packed)) Fat16Entry;

Now that we are in the right position, reading out the entries (there are bs.root_dir_entries of them) is really straightforward:

for(i=0; i<bs.root_dir_entries; i++) {
    fread(&entry, sizeof(entry), 1, in);
    print_file_info(&entry);
}

Note that I’m using a helper function for printing out the file information, as the first character of filename has a special meaning and modification time and date fields are bit-packed. See the source file for details: It’s nothing difficult, and if in doubt, you can refer to the clear explanations given for those fields in Phobos’ tutorial. I saved everything in read_root.c, compiled and ran it to see if it worked:

Amazing. If you compare this information with the screen capture found in the beginning of this tutorial, we can safely conclude that we are doing at least as good a job as Windows Explorer!

A peek into the future

Now that we have read the root directory, we are basically in the beginning of data area of the file system. Let’s try to find the first sector of README.TXT using our hex editor. Before we do that, we need to know that in FAT, the data area is divided into file allocation units called “clusters”, where each cluster is bs.sectors_per_cluster sectors long. This is because FAT uses 16-bit numbering for clusters and if each cluster was just 512 bytes long, only 32 megabytes of data could be addressed – not quite enough for a 1 GB memory card!

From read_root.exe output we can see that data area starts at 0×50200 and README.TXT is stored in cluster 0xE (14). Clusters are numbered from 2 onwards (the reason for which we’ll learn later), so to reach it, we need to skip 12 clusters (14-2) forward. Based on the output of read_boot.exe, we see that there are 32 sectors per cluster and sector size is 512 (this is 16 kiB, not coincidentally the same value I set as allocation unit size when formatting the SD card), so this is exactly 12*32*512 bytes or 0×30000 in hex. Firing up my hex editor, I opened the test.img and used the “Go to” (Ctrl-G) method to locate address 0×80200 (0×50200 + 0×30000):

How cool is that? If you only need to read a maximum of 16 kilobytes, this is all the code you will need to conquer the FAT16 file system. In the next part of this tutorial, I’ll show how to navigate the FAT to read files of any size, and maybe expand to writing files or the differences of FAT32. The parts after that will show how to interface with the SD card using a microcontroller, and adapt the code used here for the limited memory of 8-bit AVR chips. So stay tuned, and if you haven’t subscribed to the feed already, do it now!

Remark on progressive coding

In this tutorial, I’ve used the same method of “building upon the previous iteration” I use myself when tackling larger projects: I start with something really simple, and only once I get that working, I proceed to refine the code for the next step. This way, the amount of information one needs to absorb stays smaller, and you can find problems easier. If you haven’t adopted this way of developing already, I warmly recommend it!

The same method can also be applied to any electronics project: Instead of building a IR-controlled egg cooker in one go, start by trying to control the cooker with a manual switch on a breadboard. Then build a version which uses a microcontroller to toggle the switch. Then build the IR-receiver, possibly on a separate breadboard. Only once you’ve mastered each subject individually, should you combine everything together. That way you’ll waste less eggs!

Proceed to the next part of this tutorial

30 Comments or trackbacks to Simple FAT and SD Tutorial Part 1

BJ:
April 27, 2012 at 8:51

Excellent tutorial, im looking forward to the 4th installment. You may need to check this page though as I believe you have a typo “partition table should start at offset 0x1CE”, i think it is 0x1BE as in the example code.

Your teaching style is great and hope to see more.

Regards,
BJ

reply

jokkebk says:
April 27, 2012 at 9:44

Thanks for pointing it out. Fixed!

reply

Mike:
March 10, 2013 at 6:28

Interesting page, but I think your information is a little off. Not all SD card volumes will have a partition. See this page for more thorough coverage: http://www.maverick-os.dk/FileSystemFormats/FAT16_FileSystem.html

reply

jokkebk says:
March 10, 2013 at 11:03

Yes, SD cards can also be formatted in a different way, but all the cards I had followed the structure I’m describing in the tutorial, so I skipped the more advanced details for simplicity. :) Thanks for the link, it’s a useful addition to the post!

reply

roger:
June 7, 2013 at 0:39

hello, when i see the last sector that occupies the HEMLET.TXT (cluster 13), theres is some finland text or something,which is at offset 0x7F600. Why could it be there??
Thanks a lot.

reply

jokkebk says:
June 9, 2013 at 14:36

Nice find. :) I did some testing with the disc image before deciding what to write there, and as FAT doesn’t remove old data, just marks sectors free, I guess it was still there.

So if I had used zero-formatted image all unused data would be zeroes, but now there’s some old data in unused area of the sector.

reply

SM:
June 20, 2013 at 3:03

Nice one. I’m hacking on ext3, though good information.

Note: Link is dead in:


I now advise you to read the chapter “FAT and the Boot Sector” from CompuPhase tutorial.

reply

Joonas Pihlajamaa says:
December 10, 2013 at 10:21

Thanks for pointing that out. I actually forgot http:// from the beginning so the link was just broken. :)

reply

Cristi:
August 18, 2013 at 15:06

I’m confused. Removable drives don’t have MBR, they only have a single PBR. Why do you have a MBR on a removable drive?

reply

jokkebk says:
August 18, 2013 at 15:40

At least according to http://www.compuphase.com/mbr_fat.htm many CF and SD cards are arranged similarly as hard disks, and do contain a MBR. It’s been 18 months since I read more about the subject and I’ve forgotten most of the details, but at least the SD cards I had did follow the structure outlined in the tutorial. I think USB flash sticks and possibly some SD cards could be simpler.

reply

Cristi says:
August 19, 2013 at 21:50

http://en.wikipedia.org/wiki/Master_boot_record
“MBRs are not present on non-partitioned media like floppies, superfloppies or other storage devices configured to behave as such.”
I use a USB microSD card reader to format the microSD, that is basically seen as a “Removable Disk”, and all “Removable Disks” that I have don’t hold a MBR. BOOTICE also confirms this by having the button “Process MBR” grayed out and the hex editor because the first sector starts with PBR.
Anyway, is still a great tutorial.
Thanks!

reply

jokkebk says:
August 20, 2013 at 10:01

So if you make a raw copy of the SD card, the first sector actually starts with stuff that is described in “Reading and interpreting the boot sector”? Anyways, I’ll add a note to my MBR section, that there are (micro)SD cards that don’t have it at all – I did encounter it when doing research for the tutorial, but as all my SD cards had MBR, didn’t remember to mention it at all. Thanks for the information!

reply

Cristi says:
August 20, 2013 at 10:32

Yes, the first sector on my microSD is the boot sector or PBR.
Further digging showed that the presence of the MBR actually depends on who formatted the disk.
Windows likes to format removable disks as USB-FDD, which doesn’t hold a MBR with a partition table and can have only one partition.
However, using a third party software on Windows (I used BOOTICE 1.1.0), you can format the removable disk as a USB-ZIP or USB-HDD, which does hold a MBR, and can have multiple partitions.

Vladimir:
August 29, 2013 at 19:45

I think there is a contradiction here.
Boot sector is 512 bytes long. Starting from 0×00 to 0x1FF.

The Master Boot Record is also a 512 byte sector.
This sector contains a partition table with four entries starting at offset 446 (hex. 0x1BE).

How come ? 0x1be is within boot sector.
0×00 < 0x1BE < 0x1FF.

reply

Joonas Pihlajamaa says:
August 29, 2013 at 21:17

No I think it is consistent: If the SD is formatted like a hard disk with many partitions, its first sector (0) is the MBR, and partition table starts at offset 0x1BE. If there are no reserved sectors between MBR and first partition, the first partition’s boot sector is next (sector 1), also 512 bytes.

So 0x1BE is a relative offset in sector 0 (MBR). Boot sector (sector 1 or further) has different data at 0x1BE of that sector.

What may be confusing is the sector,offset method of addressing stuff instead of just linear offset – if you’re viewing a disk image in hex editor, sector X offset 0x1BE is at X*0×200 + 0x1BE (for example 0x3BE for sector 1). As a car analogy, you could think a sector as a bus with 512 seats. First bus (MBR) seat 0x1BE is the beginning of partition table, but there is a seat 0x1BE in every bus after that, including the boot sector which is usually the second bus, right after MBR. :)

reply

vladimir says:
August 30, 2013 at 17:23

It is OK now. The problem was HxD software.
When opened SD card it showed me all data starting from first boot sector af active partition.
Now I use WinHex. It is OK. MBR is at sector 0, and boor record is at xxx butes after.

reply

Joonas Pihlajamaa says:
September 2, 2013 at 9:44

Yeah, I think also that if not running as administrator, HxD doesn’t even offer the option to read whole raw device, just the partitions.

reply

Sebastian:
December 3, 2013 at 5:57

im a noob trying here. I got the first exempel successfully compiled and it printed out the Partition Tabel just fine. Im using the test.img you provided.

But when i printing out the boot sector it dosent match with the image you are showing. Jump code and OEM code match, but after that it derails.

Is it my compiler somehow, or is the test.img diffrent from the test.img printed on the read_boot.png image?

Thanks for great tutorial anyways!

reply

Joonas Pihlajamaa says:
December 9, 2013 at 23:12

The compiler might be the issue, e.g. different size for short or not respecting the __packed__ attribute (i.e. don’t pad fields to 32-bit boundaries). You could print out the field offsets with something like:

printf(“Attributes at %d\n”, (void *)bs.attributes – (void *)bs);

Not quite sure about the casting but a substraction and proper cast should do the trick.

I used gcc from MinGW project myself, what platform and compiler you are using?

reply

Sebastian says:
December 11, 2013 at 5:12

Yes, you are right. After some digging around i found this:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=52991

The last comment said there was a patch for this for some time now, but i dont know where i can get it, or apply it or anything.

So i almost quite. But some fooling around i found some guy do this: #define PACKED __attribute__ ((__packed__))

And after applying it to some of the structor members, ie:
PACKED unsigned short sector_size;

I got it to match the result you are showning. So im reading thru this tutorial now :)

reply

Sandeep Tayal:
January 10, 2014 at 8:41

Really Nice Tutorial. I liked your iterative approach to develop projects. I will adapt that.

reply

Miron:
February 1, 2014 at 7:23

Good job! Exactly what I need! Thanks a lot for full description of this material!Good luck for your posting!

reply

Leave a Reply

Your e-mail address will not be published.


× four = 20

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>