Writing a custom Filesystem Format
CodeGuru Home VC++ / MFC / C++ .NET / C# Visual Basic VB Forums Developer.com
Results 1 to 11 of 11

Thread: Writing a custom Filesystem Format

  1. #1
    Join Date
    Jan 2011
    Posts
    9

    Question Writing a custom Filesystem Format

    There is one thing that I've always wondered about, and yet never seem to find when searching through google, tutorials, or even looking at programming textbooks.
    The question seems easy enough, but the answer seems elusive.

    How do you write your own Filesystem/Format?

    I've always wanted to know how to write my own system for the learning experience.
    I've worked on simple ones that read from a flatfile of a defined size, but never one that was on a fully formatted drive.

    Where do I start? Are there any examples/tutorials?

  2. #2
    Join Date
    Jun 2009
    Location
    France
    Posts
    2,286

    Re: Writing a custom Filesystem Format

    You don't "write" a filesystem format, you define it. That's the "easy" part.

    The hard part is getting the OS to recognize it and interface with it. This requires, either:
    - A third party program that accesses the drives directly
    - A driver for the OS
    - Kernal inclusion

    Needless to say, all of the above are far from trivial, even for professionals, and well outside the scope of plain old C++.

    To put things into perspective, NTFS is still not fully supported on linux type systems. ext2 is supposedly supported on windows with thrid party drivers, but ext3 and ext4 aren't.

    And I can tell you there is a LOT of pressure to get those working.
    Is your question related to IO?
    Read this C++ FAQ LITE article at parashift by Marshall Cline. In particular points 1-6.
    It will explain how to correctly deal with IO, how to validate input, and why you shouldn't count on "while(!in.eof())". And it always makes for excellent reading.

  3. #3
    Join Date
    Apr 2012
    Posts
    5

    Re: Writing a custom Filesystem Format

    Actually the problem with NTFS is that the documentation on the spec was never supposed to become public. Bits and pieces leaked out over the years; whether or not Microsoft has finally just released the entire format is not something I've kept up to date on. I had to write code to pull the paging file off the disk (and what I found in there would make your skin crawl!!!).

  4. #4
    Join Date
    Jan 2011
    Posts
    9

    Re: Writing a custom Filesystem Format

    Quote Originally Posted by monarch_dodra View Post
    You don't "write" a filesystem format, you define it. That's the "easy" part.

    The hard part is getting the OS to recognize it and interface with it. This requires, either:
    - A third party program that accesses the drives directly
    - A driver for the OS
    - Kernal inclusion

    Needless to say, all of the above are far from trivial, even for professionals, and well outside the scope of plain old C++.

    To put things into perspective, NTFS is still not fully supported on linux type systems. ext2 is supposedly supported on windows with thrid party drivers, but ext3 and ext4 aren't.

    And I can tell you there is a LOT of pressure to get those working.
    I'm not referring to the likes of NTFS because, as CMalcheski said, it's a proprietary Microsoft filesystem that wasn't meant to become public.

    I meant something more-or-less basic for testing. I'm just asking mainly about how to format a drive in a custom format.
    I don't mean the OS drive either, just -- for example -- a USB key, or another partition.

    I know it'd require different applications to access/alter/read/write information to it already.
    I'm not saying to entirely incorporate it into the system and make it work with the basic file manager, I'm saying to build a separate one that will work with the new partition.

  5. #5
    Join Date
    Oct 2006
    Location
    Sweden
    Posts
    3,632

    Re: Writing a custom Filesystem Format

    Go ahead and make something of you own mind but it doesn't matter, you still have to do what monarch_dodra says.
    Debugging is twice as hard as writing the code in the first place.
    Therefore, if you write the code as cleverly as possible, you are, by
    definition, not smart enough to debug it.
    - Brian W. Kernighan

    To enhance your chance's of getting an answer be sure to read
    http://www.codeguru.com/forum/announ...nouncementid=6
    and http://www.codeguru.com/forum/showthread.php?t=366302 before posting

    Refresh your memory on formatting tags here
    http://www.codeguru.com/forum/misc.php?do=bbcode

    Get your free MS compiler here
    http://www.microsoft.com/visualstudio/eng/downloads

  6. #6
    Join Date
    Jan 2011
    Posts
    9

    Re: Writing a custom Filesystem Format

    Quote Originally Posted by S_M_A View Post
    Go ahead and make something of you own mind but it doesn't matter, you still have to do what monarch_dodra says.
    I'm not attempting to dispute what he's saying, just trying to understand it, which I actually get now.
    At first I misunderstood what he meant, thinking that drivers would be to integrate the system into the OS's file manager -- but then I realized that's to recognize the actually drive/format, and detect how the file system is dealt with before data manipulation can even be achieved.

    I realize that creation of one, in a whole, is not particularly easy by any means -- but it's far from unachievable. I'm just merely trying to learn, that's all. No disrespect intended.

  7. #7
    Join Date
    Jan 2011
    Posts
    9

    Re: Writing a custom Filesystem Format

    By chance, searching for how to write my own device driver lead my to the Windows Driver Kit, which includes a built in API for recognizing file systems.

    I wasn't looking specifically for Windows, but I think I'll take a look there.

  8. #8
    Join Date
    Mar 2010
    Location
    Melbourne Australia
    Posts
    454

    Re: Writing a custom Filesystem Format

    At Amazon search for a book Linux kernel development (3rd Edition ) , on page 261 page the book you will find virtual filing system , ( all filing systems under linux are implemented as such ) Study the interface and implementation ( language C ) after you are mastered this ( you can even submit some patches ) you are then in a position to develop your own. Writing a good filing system as hard as developing a kernel ( operating system ). Lot of things have to be mastered , like data structures in C, As you would Digg deep you will find only the best of developers get to work on these.

  9. #9
    Join Date
    Jun 2010
    Location
    Germany
    Posts
    2,583

    Arrow Re: Writing a custom Filesystem Format

    This may be some interesting related reading too: http://wiki.osdev.org/File_Systems, including links to more stuf.
    I was thrown out of college for cheating on the metaphysics exam; I looked into the soul of the boy sitting next to me.

    This is a snakeskin jacket! And for me it's a symbol of my individuality, and my belief... in personal freedom.

  10. #10
    Join Date
    Jan 2011
    Posts
    9

    Re: Writing a custom Filesystem Format

    Quote Originally Posted by aamir121a View Post
    At Amazon search for a book Linux kernel development (3rd Edition ) , on page 261 page the book you will find virtual filing system , ( all filing systems under linux are implemented as such ) Study the interface and implementation ( language C ) after you are mastered this ( you can even submit some patches ) you are then in a position to develop your own. Writing a good filing system as hard as developing a kernel ( operating system ). Lot of things have to be mastered , like data structures in C, As you would Digg deep you will find only the best of developers get to work on these.
    Thank you so much for that! Seriously, I wasn't aware of the book's existence, the only book I really have is Stroustrup's book the C++ Programming Language.

    I love learning about kernal/file system development. I much prefer designing and working on the back end of systems

  11. #11
    Join Date
    Jul 2012
    Posts
    1

    Re: Writing a custom Filesystem Format

    I signed up to respond to this question...

    Check out Fuse. It allows you to slap together your own FS quickly and easily. http://fuse.sourceforge.net/
    Example: http://fuse.sourceforge.net/helloworld.html
    You will need to make sure you map your custom functions like so:
    Code:
    static struct fuse_operations hello_oper = {
        .getattr	= hello_getattr,
        .readdir	= hello_readdir,
        .open	= hello_open,
        .read	= hello_read,
    };
    Also, if you want to get your hands dirty, as aamir121a said, the linux kernel development is a good place to look. As you have already discovered, there isn't many web resources for this topic, but if you download the linux kernel source and look at ../fs/ramfs you will see one of the smallest FSs implemented directly into the kernel.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  


Azure Activities Information Page

Windows Mobile Development Center


Click Here to Expand Forum to Full Width

This is a CodeGuru survey question.


Featured


HTML5 Development Center