Simple FAT and SD Tutorial Part 1
Are you limited by 128 bytes of EEPROM on your MCU or even the few kilobytes of flash in your project? Instead of just downloading a library like Petit FAT File System Module and following blindly a tutorial on how to customize it to your microcontroller and SD card, would you like to really understand what you are doing, and maybe learn a bit about filesystems and SPI in the process?
In this first part of my FAT and SD tutorial, we’ll take a SD card image, and create a simple C program to interpret its contents. For this part, you don’t need any hardware at all, just a computer with gcc (GNU C Compiler) or any other ANSI C compatible compiler installed.
Getting ready: Hex editor and disk image
To make the coding easier, I recommend a good hex editor. The one I’m using is the free and excellent HxD by Maël Hörz. You can also use it to create a 1:1 disk image from a physical SD card. To have a filesystem to read, I purchased a 1 GB micro-SD card with SD adapter for 5€, plugged it into my computer and formatted it as FAT16 (over 2 GB cards will likely get formatted as FAT32), and copied Hamlet from Project Gutenberg and some other dummy test files to it (also created a subdirectory with a few text files in it):
I then launched HxD in Administrator mode to allow it to open physical disks. Note that you don’t open just the one FAT16 partition (“H:”), but the whole removable disk (I circled the icon you need to click first):
Because the partition data, file system and few test files only occupy only the very beginning the 1 GB disk, I then proceeded to select only the first 1 MB (Edit > Select block… > Offset 0-FFFFF) and copy-pasted that into a new file, which I then saved as test.img in my project folder. If you don’t have a smallish SD/micro-SD card at hand, you can just grab the project zip and use my test image.
Note: If you’re using Linux or a Mac, you can just use dd if=/dev/sda of=~/test.img bs=1M count=1 to create the image (assuming the kernel selected /dev/sda as the device for your card). You may need to add “sudo” in the beginning.
Reading the partition data
Now that we have a file image of the SD card (or the beginning of it), we can start poking around. To get going, I first read the FAT on [SD cards] tutorial by Thiadmer Riemersma and Claudio Toffoli of CompuPhase, and I really suggest you to read the 1 page of text under the heading “The Master Boot Record and the partition table” at this point. In short, we expect the test.img to be roughly laid out like this:
- MBR and partition table
- …maybe some sectors for logical partitions or reserved space…
- Boot sector of a FAT16 partition
- …optionally some reserved sectors…
- 1-2 copies of File Allocation Table (FAT)
- Root directory
- Other directories and data
Update 2013-08-20:: As one of my readers pointed out, some SD/micro-SD cards don’t have several partitions and MBR at all, but are formatted like floppy disks of old. These types of media start straight with a boot sector, described in next section.
Before we go further, you should know that the basic storage block in filesystems is called a sector, and that is simply a 512-byte (256-word) block of data. The MBR and partition table take up one sector. Boot sector will take up one sector. The copies of FAT take some N sectors, and so on. Diskettes and older operating systems used a so-called CHS (cylinder, head, sector) addressing where the sector was the smallest unit, but we don’t need to know about cylinders and heads, just the sector will do for now.
From the tutorial I linked above, we learn that the partition table should start at offset 0x1BE (hex numbers are used a lot) and contain four 16-byte partition entries, each laid out in the following manner:
| Offset | Description | Size |
|---|---|---|
| 0 | 0x80 if active (bootable), 0 otherwise | 1 |
| 1 | Start of the partition in CHS-addressing | 3 |
| 4 | Type of the partition | 1 |
| 5 | End of the partition in CHS-addressing | 3 |
| 8 | Relative offset to the partition in sectors (LBA) | 4 |
| 12 | Size of the partition in sectors | 4 |
The partition type should be 4 for FAT16 partitions that are less than 32 MiB (MiB = 1024^2 bytes), 6 for over 32 MiB partitions and 14 for FAT16 partitions using LBA addressing. Let’s try reading the partition table in C:
#include <stdio.h>
#include <stdlib.h>
int main() {
FILE * in = fopen("test.img", "rb");
unsigned int i, start_sector, length_sectors;
fseek(in, 0x1BE, SEEK_SET); // go to partition table start
for(i = 0; i < 4; i++) { // read all four entries
printf("Partition entry %d: First byte %02X\n", i, fgetc(in));
printf(" Partition start in CHS: %02X:%02X:%02X\n", fgetc(in), fgetc(in), fgetc(in));
printf(" Partition type %02X\n", fgetc(in));
printf(" Partition end in CHS: %02X:%02X:%02X\n", fgetc(in), fgetc(in), fgetc(in));
fread(&start_sector, 4, 1, in);
fread(&length_sectors, 4, 1, in);
printf(" Relative LBA address %08X, %d sectors long\n", start_sector, length_sectors);
}
fclose(in);
return 0;
}Save this as read_mbr.c in the same folder that you saved test.img, and compile with gcc read_mbr.c -o read_mbr.exe and run:
Pretty easy, huh! But reading every field manually gets cumbersome after a while, so we could just define everything in a nice struct. Here’s the modified read_mbr2.c:
#include <stdio.h>
#include <stdlib.h>
typedef struct {
unsigned char first_byte;
unsigned char start_chs[3];
unsigned char partition_type;
unsigned char end_chs[3];
unsigned long start_sector;
unsigned long length_sectors;
} __attribute((packed)) PartitionTable;
int main() {
FILE * in = fopen("test.img", "rb");
int i;
PartitionTable pt[4];
fseek(in, 0x1BE, SEEK_SET); // go to partition table start
fread(pt, sizeof(PartitionTable), 4, in); // read all four entries
for(i=0; i<4; i++) {
printf("Partition %d, type %02X\n", i, pt[i].partition_type);
printf(" Start sector %08X, %d sectors long\n",
pt[i].start_sector, pt[i].length_sectors);
}
fclose(in);
return 0;
}Note that I printed out only the minimum needed information this time. The __attribute((packed)) is needed so the compiler won’t align our data structure fields to 32-bit or 64-bit boundaries (by default, it likes to pad structures with empty areas so the processor can handle the data a bit faster) and make a mess. Also, I’m using a compiler where sizeof(char) is 1, sizeof(short) is 2 and sizeof(long) is 4, adjust your data types if you’re using a different system or compiler!
Reading and interpreting the boot sector
Based on the previous output, it is rather evident that the first partition is the one we want to examine further. I now advise you to read the chapter “FAT and the Boot Sector” from CompuPhase tutorial. Based on the tutorial, the boot sector for a FAT16 file system should nicely fit into this structure:
typedef struct {
unsigned char jmp[3];
char oem[8];
unsigned short sector_size;
unsigned char sectors_per_cluster;
unsigned short reserved_sectors;
unsigned char number_of_fats;
unsigned short root_dir_entries;
unsigned short total_sectors_short; // if zero, later field is used
unsigned char media_descriptor;
unsigned short fat_size_sectors;
unsigned short sectors_per_track;
unsigned short number_of_heads;
unsigned long hidden_sectors;
unsigned long total_sectors_long;
unsigned char drive_number;
unsigned char current_head;
unsigned char boot_signature;
unsigned long volume_id;
char volume_label[11];
char fs_type[8];
char boot_code[448];
unsigned short boot_sector_signature;
} __attribute((packed)) Fat16BootSector;
To make the code a bit more robust, I used a for loop that stops when it finds the first partition with compatible partition type (4, 6 or 14), so the partition entry is in pt[i]. To move inside the image file, we’ll first use the fseek() command, then read the boot sector:
fseek(in, 512 * pt[i].start_sector, SEEK_SET);
fread(&bs, sizeof(Fat16BootSector), 1, in);I added a few printout commands and saved the result into read_boot.c – you should understand it easily when comparing with read_mbr2.c. The results of a test run can be seen on the right.
Reading the root directory
Before we dive deep into the file allocation table, we’ll peek ahead and read the root directory. To do that, we need to skip over any reserved sectors (amount of reserved sectors is bs.reserved_sectors, which includes the boot sector we just read) and all FATs (there are bs.number_of_fats of them, each bs.fat_size_sectors sectors long):
fseek(in, (bs.reserved_sectors-1 + bs.fat_size_sectors * bs.number_of_fats) *
bs.sector_size, SEEK_CUR);Note that we’re using the SEEK_CUR seek mode, which uses the current location as a base. This statement is added after reading the boot sector, which is why we compensate for it by substracting one from bs.reserved_sectors.
For this part, I’m using the Phobos FAT file system tutorial as a reference. It’s more detailed on the FAT part than the previous tutorial. “The Root Directory” chapter contains the information we need to decipher the structure of file entries:
typedef struct {
unsigned char filename[8];
unsigned char ext[3];
unsigned char attributes;
unsigned char reserved[10];
unsigned short modify_time;
unsigned short modify_date;
unsigned short starting_cluster;
unsigned long file_size;
} __attribute((packed)) Fat16Entry;Now that we are in the right position, reading out the entries (there are bs.root_dir_entries of them) is really straightforward:
for(i=0; iNote that I'm using a helper function for printing out the file information, as the first character of filename has a special meaning and modification time and date fields are bit-packed. See the source file for details: It's nothing difficult, and if in doubt, you can refer to the clear explanations given for those fields in Phobos' tutorial. I saved everything in read_root.c, compiled and ran it to see if it worked:
Amazing. If you compare this information with the screen capture found in the beginning of this tutorial, we can safely conclude that we are doing at least as good a job as Windows Explorer!
A peek into the future
Now that we have read the root directory, we are basically in the beginning of data area of the file system. Let's try to find the first sector of README.TXT using our hex editor. Before we do that, we need to know that in FAT, the data area is divided into file allocation units called "clusters", where each cluster is bs.sectors_per_cluster sectors long. This is because FAT uses 16-bit numbering for clusters and if each cluster was just 512 bytes long, only 32 megabytes of data could be addressed - not quite enough for a 1 GB memory card!
From read_root.exe output we can see that data area starts at 0x50200 and README.TXT is stored in cluster 0xE (14). Clusters are numbered from 2 onwards (the reason for which we'll learn later), so to reach it, we need to skip 12 clusters (14-2) forward. Based on the output of read_boot.exe, we see that there are 32 sectors per cluster and sector size is 512 (this is 16 kiB, not coincidentally the same value I set as allocation unit size when formatting the SD card), so this is exactly 1232512 bytes or 0x30000 in hex. Firing up my hex editor, I opened the test.img and used the "Go to" (Ctrl-G) method to locate address 0x80200 (0x50200 + 0x30000):
How cool is that? If you only need to read a maximum of 16 kilobytes, this is all the code you will need to conquer the FAT16 file system. In the next part of this tutorial, I'll show how to navigate the FAT to read files of any size, and maybe expand to writing files or the differences of FAT32. The parts after that will show how to interface with the SD card using a microcontroller, and adapt the code used here for the limited memory of 8-bit AVR chips. So stay tuned, and if you haven't subscribed to the feed already, do it now!
Remark on progressive coding
In this tutorial, I've used the same method of "building upon the previous iteration" I use myself when tackling larger projects: I start with something really simple, and only once I get that working, I proceed to refine the code for the next step. This way, the amount of information one needs to absorb stays smaller, and you can find problems easier. If you haven't adopted this way of developing already, I warmly recommend it!
The same method can also be applied to any electronics project: Instead of building a IR-controlled egg cooker in one go, start by trying to control the cooker with a manual switch on a breadboard. Then build a version which uses a microcontroller to toggle the switch. Then build the IR-receiver, possibly on a separate breadboard. Only once you've mastered each subject individually, should you combine everything together. That way you'll waste less eggs!