Writing my own standard-compatible file buffer
CodeGuru Home VC++ / MFC / C++ .NET / C# Visual Basic VB Forums Developer.com
Results 1 to 10 of 10

Thread: Writing my own standard-compatible file buffer

Hybrid View

  1. #1
    Join Date
    Feb 2009
    Posts
    5

    Question Writing my own standard-compatible file buffer

    Hi,

    I am trying to implement a new type similar to std::ifstream which works in an MPI-parallel environment. My approach is to write a dedicated buffer which inherits from std::streambuf; then, I write a dedicated file-stream class which inherits from from std::istream and uses the dedicated buffer.

    My implementation works with the gcc compiler, it works with the Intel compiler, but it doesn't work with the Portland compiler (it always reads zeros when reading an input file). I suspect that my implementation is not fully standard compatible, and I wonder if somebody can help me with this issue. Here is a minimal version of my code, which does NOT work with Portland. This minimal version doesn't do any parallelism. It basically encapsulates a std::streambuf object by fowarding all function calls to the original std::streambuf. This is kind of trivial, and I find myself unable to figure out what's wrong here.

    Thank you in advance for your help.

    Code:
    class ParBuf : public std::streambuf {
    public:
      ParBuf(std::streambuf* originalBuf_);
    protected:
      virtual int_type overflow (int_type c);
      virtual std::streamsize xsputn(const char* s, std::streamsize num);
    
      virtual int_type uflow();
      virtual int_type underflow();
      virtual std::streamsize xsgetn (char* s, std::streamsize num);
    private:
      std::streambuf* originalBuf;
    };
    
    class my_ifstream : public std::istream {
    public:
      my_ifstream();
      explicit my_ifstream(const char * filename,
                           openmode mode = in );
      ~my_ifstream();
    
      std::streambuf* rdbuf() const;
      bool is_open();
      void open(const char* filename, openmode mode = in);
      void close();
    private:
      std::filebuf   fbuf;
      mutable ParBuf mybuf;
    };
    
    ParBuf::ParBuf(std::streambuf* originalBuf_)
      : originalBuf(originalBuf_)
    { }
    
    std::streambuf::int_type
    ParBuf::overflow (std::streambuf::int_type c) {
        int_type returnVal = c;
        if (c != EOF) {
            returnVal = originalBuf->sputc((char)c);
        }
        return returnVal;
    }
    
    std::streamsize
    ParBuf::xsputn(const char* s, std::streamsize num) {
        return originalBuf->sputn(s,num);
    }
    
    std::streambuf::int_type
    ParBuf::uflow() {
        int_type = originalBuf->sbumpc();
        return value;
    }
    
    std::streambuf::int_type
    ParBuf::underflow() {
        int_type value  = originalBuf->sgetc();
        return value;
    }
    
    std::streamsize
    ParBuf::xsgetn (char* s, std::streamsize num) {
        std::streamsize sizeRead = originalBuf->sgetn(s, num);
        return sizeRead;
    }
    
    my_ifstream::my_ifstream() : std::istream(NULL), fbuf(), mybuf(&fbuf) {
        init(&mybuf);
    }
    
    my_ifstream::my_ifstream(const char * filename, openmode mode)
        : std::istream(NULL), fbuf(), mybuf(&fbuf)
    {
        init(&mybuf);
        open(filename, mode);
    }
    
    my_ifstream::~my_ifstream()
    { }
    
    std::streambuf*
    my_ifstream::rdbuf() const {
        return &mybuf;
    }
    
    bool my_ifstream::is_open() {
        return fbuf.is_open();
    }
    
    void my_ifstream::open(const char* filename, openmode mode)
    {
        int ok = (bool) fbuf.open(filename, mode | ios_base::in);
        if (!ok) {
            this->setstate(ios_base::failbit);
        }
    }
    
    void my_ifstream::close() {
        int ok = (bool) fbuf.close();
        if (!ok) {
            setstate(ios_base::failbit);
        }
    }
    Last edited by signalingNaN; February 8th, 2009 at 12:09 PM.

  2. #2
    Join Date
    Nov 2003
    Posts
    1,825

    Re: Writing my own standard-compatible file buffer

    Please edit your post and re-post your code using code-tags (making it readable) - http://www.codeguru.com/forum/misc.php?do=bbcode#code

    >> I am trying to implement a new type similar to std::ifstream which works in an MPI-parallel environment.
    Out of curiosity, what do these "new types" actually do? What purpose do they server - or try to accomplish?

    gg

  3. #3
    Join Date
    Feb 2009
    Posts
    5

    Re: Writing my own standard-compatible file buffer

    I'm in working in a distributed-memory environment in which multiple threads are executed concurrently and communicate through a message-passing interface (MPI). The programming model is Single-Program-Multiple-Data (SPMD) in which all threads execute the same code. No here's the thing: only one thread has access to the hard disk, but whithin this programming model, one pretends that everyone has access (because they all do the same thing).

    My modified file stream offers the required abstraction. It reads data on one thread only and then communicates the data to the other threads so that they can behave as if they had access to the disk. For simplicity, I have eliminated the MPI communication and produced a minimal example which works on Intel/GCC but not on Portland. My minimal example does nothing else than redefine the existing std::ifstream. I guess that if we find out what's wrong with this example I am able to correct the full MPI case.

    Thanks in advance.

  4. #4
    Join Date
    Nov 2003
    Posts
    1,825

    Re: Writing my own standard-compatible file buffer

    It would be easier if you provided a main() and #include's - basically something complete that should compile.
    Then using that code, copy/paste the errors that Portland is giving you.

    gg

  5. #5
    Join Date
    Feb 2009
    Posts
    5

    Re: Writing my own standard-compatible file buffer

    You are right; excuse me for being sloppy. So, here's a fully featured code which compiles and which does something: read an integer value from an ASCII file and print it. With Portland, there's no error message. But instead of printing the expected value, it prints "0".

    My version of the Portland compiler is pgCC 7.2-5 64-bit.

    Thanks.

    Code:
    #include <streambuf>
    #include <istream>
    #include <ostream>
    #include <fstream>
    #include <iostream>
    
    class ParBuf : public std::streambuf {
    public:
      ParBuf(std::streambuf* originalBuf_);
    protected:
      virtual int_type overflow (int_type c);
      virtual std::streamsize xsputn(const char* s, std::streamsize num);
    
      virtual int_type uflow();
      virtual int_type underflow();
      virtual std::streamsize xsgetn (char* s, std::streamsize num);
    private:
      std::streambuf* originalBuf;
    };
    
    class my_ifstream : public std::istream {
    public:
      my_ifstream();
      explicit my_ifstream(const char * filename,
                           openmode mode = in );
      ~my_ifstream();
    
      std::streambuf* rdbuf() const;
      bool is_open();
      void open(const char* filename, openmode mode = in);
      void close();
    private:
      std::filebuf   fbuf;
      mutable ParBuf mybuf;
    };
    
    ParBuf::ParBuf(std::streambuf* originalBuf_)
      : originalBuf(originalBuf_)
    { }
    
    std::streambuf::int_type
    ParBuf::overflow (std::streambuf::int_type c) {
        int_type returnVal = c;
        if (c != EOF) {
            returnVal = originalBuf->sputc((char)c);
        }
        return returnVal;
    }
    
    std::streamsize
    ParBuf::xsputn(const char* s, std::streamsize num) {
        return originalBuf->sputn(s,num);
    }
    
    std::streambuf::int_type
    ParBuf::uflow() {
        int_type value = originalBuf->sbumpc();
        return value;
    }
    
    std::streambuf::int_type
    ParBuf::underflow() {
        int_type value  = originalBuf->sgetc();
        return value;
    }
    
    std::streamsize
    ParBuf::xsgetn (char* s, std::streamsize num) {
        std::streamsize sizeRead = originalBuf->sgetn(s, num);
        return sizeRead;
    }
    
    my_ifstream::my_ifstream() : std::istream(NULL), fbuf(), mybuf(&fbuf) {
        init(&mybuf);
    }
    
    my_ifstream::my_ifstream(const char * filename, openmode mode)
        : std::istream(NULL), fbuf(), mybuf(&fbuf)
    {
        init(&mybuf);
        open(filename, mode);
    }
    
    my_ifstream::~my_ifstream()
    { }
    
    std::streambuf*
    my_ifstream::rdbuf() const {
        return &mybuf;
    }
    
    bool my_ifstream::is_open() {
        return fbuf.is_open();
    }
    
    void my_ifstream::open(const char* filename, openmode mode)
    {
        int ok = (bool) fbuf.open(filename, mode | ios_base::in);
        if (!ok) {
            this->setstate(ios_base::failbit);
        }
    }
    
    void my_ifstream::close() {
        int ok = (bool) fbuf.close();
        if (!ok) {
            setstate(ios_base::failbit);
        }
    }
    
    int main() {
        my_ifstream ifile("someFile.dat");
        int someValue;
        // The following line reads data from "someFile.dat", assuming it contains
        // ASCII formated integers. With Intel and GCC, this works. With Portland,
        // I get the value 0 instead of the expected one.
        ifile >> someValue;
        std::cout << someValue << std::endl;
    }

  6. #6
    Join Date
    Nov 2003
    Posts
    1,825

    Re: Writing my own standard-compatible file buffer

    So ParBuf, which is a std::streambuf, is simply trying to forward calls to a member std::streambuf named "originalBuf". However, you've only forwarded 5 of the 18 (or so) public methods of std::streambuf. You're just getting lucky with the Intel and GCC implementations.

    Instead of forwarding calls, it's much easier to implement a real stream buffer. But in this case, it seems that ParBuf should be inheriting from std::filebuf - then "hook" virtual methods as needed. But I have some questions (below) before we start getting into a design...

    So what goals are you actually trying to achieve by extending standard I/O classes? Typically, a network filesystem (NFS) is used so that each MPI node can access the same set of files. But if that's not the case here, what kind of access are you trying to provide? Do you want to do something very specific like partition a single file's data among all the nodes? Or something very generic like providing NFS-like access to the full content of a file on a remote node? Or something else?

    gg

  7. #7
    Join Date
    Feb 2009
    Posts
    5

    Re: Writing my own standard-compatible file buffer

    Thanks for your help.

    The most common use of my customized file stream will be: easy way of reading input parameters. Example:

    Code:
    my_ifstream ifile("parameters.dat");
    bool useSomeLibary;
    ifile >> useSomeLibrary;
    But moreover, I'd like to offer means of reading any data file.

    Every MPI thread should end up with the same value assigned to the variable "useSomeLibrary". I understand that if the system has NFS, I can use ifstream instead of my_ifstream in the above example. But the thing is, I'd prefer to have a code which works without assumption on the file system. For some reason (user quota restrictions, efficiency, or other), I've observed that some users prefer to store their data on a local hard disk rather than NFS.

  8. #8
    Join Date
    Nov 2003
    Posts
    1,825

    Re: Writing my own standard-compatible file buffer

    Well, implementing a generic file server using MPI doesn't really make sense, since it wasn't designed for client/server, command/response style communication (but there's nothing stopping you from using it that way).

    I should also note that as of MPI-2.0, there are file access API's (but we don't have to get into that).
    http://www.mpi-forum.org/docs/mpi21-...57.htm#Node257

    If you want to represent startup parameters as a single string, that makes more sense. It's not as efficient has sending native data types (MPI_INT etc...), either as an array of a single data type, or by packing and unpacking each configuration parameter. If each parameter can be represented as a single data type (like an int), then you might as well send a single array and be done with it.

    Assuming you still want to send a file, the easiest thing to do is to have the root node read the file contents into memory, then broadcast the contents to each node. (You'll have to MPI_Bcast the size of the contents first, and then the contents itself - or use a fixed sized buffer on all nodes). Once all the nodes have the files contents, it's fairly easy to create an input stream to read the contents. For example:
    Code:
    #include <streambuf>
    
    template <class charT, class traits = std::char_traits<charT> >
    class basic_fixedbuf_reader : public std::basic_streambuf<charT, traits>
    {
    public:
        // standard streambuf types
        typedef charT                       char_type;
        typedef typename traits::int_type   int_type;
        typedef typename traits::pos_type   pos_type;
        typedef typename traits::off_type   off_type;
        typedef traits                      traits_type;
    
        basic_fixedbuf_reader(const charT *s, std::streamsize len)
        {
            // setg() is a non-const interface, but we know that no writing will
            // occur to this buffer
            charT *p = const_cast<charT*>(s);
            setg(p, p, p + len);
        }//basic_fixedbuf_reader
    };//basic_fixedbuf_reader
    
    typedef basic_fixedbuf_reader<char>     fixedbuf_reader;
    typedef basic_fixedbuf_reader<wchar_t>  wfixedbuf_reader;
    
    //------------------------------------------------------------------------------
    
    #include <iostream>
    #include <string>
    #include <cstring>
    using namespace std;
    
    int main()
    {
        const char str[] = "Hello World!";
    
        // create a custom input stream buffer to read directly from str[]
        // NOTE: fbr must have at least the same lifetime/scope as str[]
        fixedbuf_reader fbr(str, strlen(str));
    
        // create an input stream to perform formatted input from fbr (aka str[])
        istream in(&fbr);
    
        // read a string at a time, dumping to cout
        for (string s; in >> s;)
            cout << s << endl;
    
        return 0;
    }//main
    fixedbuf_reader is a simple stream buffer for reading from an external, fixed buffer. So each of the non-root nodes will receive the broadcasted file contents and construct an input stream to read those contents. (Note that I didn't use the <strstream> header - since it's deprecated.)

    gg

  9. #9
    Join Date
    Feb 2009
    Posts
    5

    Re: Writing my own standard-compatible file buffer

    Thank you for your input, Codeplug. Your idea of writing a new stream buffer right away is valuable.

    I do agree with you on your comments about using MPI- 2 functionalities for file I/O. I was probably not clear enough in my explanations; my issue was not about which technology to use to read data from a parallel file system, but rather how to create a unified user interface which encapsulates these details. The typical end-users of my software are scientists with little programming skills and, especially, little knowledge of parallelism. They do appreciate the ability to write programs which you can compile either for sequential or parallel execution, without modification. It seemed to me that C++ streams offer a nice way of implementing this abstraction. Simply write a command like
    Code:
    myFile >> myData;
    and let the right thing happen, in serial or parallel. In parallel, what's going on behind the scene may be an MPI-2 function call or not, but the issue here is the unified user interface.
    Thanks again.

  10. #10
    Join Date
    Nov 2003
    Posts
    1,825

    Re: Writing my own standard-compatible file buffer

    >> but the issue here is the unified user interface.
    The interface of standard streams I/O is well-known. The question is what semantics does this interface provide under an MPI environment. Is it just a home grown file server? Or does it encapsulate and provide similar functionality to what MPI2 file I/O provides?

    Gotta know what it does before we can talk about how to do it - or if the standard streams interface is even appropriate.

    gg

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  


Windows Mobile Development Center


Click Here to Expand Forum to Full Width

This is a CodeGuru survey question.


Featured


HTML5 Development Center