Strange memory error

[ultracow]

Hello, i've been stuck with this error for days now and I can't seem to find what is causing it. I have tried running in debug mode but that just stops on pretty much random locations and has been no help. I tried valgrind but it doesnt seem to give me any idea of where the error is occuring. The error happens in my image decoding class:

/* Decoder klasse gemaakt door Ben Balthazar */
#include "decoder.h"
#include <fstream>
#include "IOException.h"
#include <QGenericMatrix>
using namespace std;
Decoder::Decoder(const Settings &s)
{
    this->set = s;

}

QMatrix4x4 Decoder::inverseDCT(QMatrix4x4 y) {
    QMatrix4x4 result;
    result = set.DCT().transposed() * y;
    result = result * set.DCT();
    return result;
}

uint8_t* Decoder::ReadFile() {
    ifstream in;
    in.open(set.EncFile().toLatin1(),ios_base::binary);
    in.seekg(0,ios_base::end);
    int filesize = in.tellg();
    in.seekg(0,ios_base::beg);
    char *input = new char[filesize];
    uint8_t* buff = (uint8_t*) malloc(filesize*sizeof(char));
    in.read(input,filesize);
    in.close();
    for(int i=0;i<filesize;++i) {
        buff[i] = input[i];
    }
    free(input);

    return buff;
}

util::BitStreamWriter Decoder::Decode(uint8_t* buff,int size) {
    int row=0,col=0;
    int **image = (int**) malloc(set.Height()*sizeof(int*));
    for(int i = 0; i < set.Height(); ++i)
    {
        image[i] = (int*)malloc(set.Width() * sizeof(int));
    }
    for(int i=0;i<set.Height();++i)
        for(int j=0;j<set.Height();++j)
            image[i][j] = 0;
    uint8_t* buffCopy = new uint8_t[size];
    for(int i=0;i<size;++i)
        buffCopy[i] = buff[i];
    util::BitStreamReader b(buffCopy,size);
    /* Start processing matrixes 1 by 1 */
    signed int* list = new int[16];
    for(int i=0;i<set.Width()*set.Height()/16;++i) {

        if(set.RLE()) {
            uint8_t* tempList = DecodeList(b);*/
            
            for(int j=0;j<16;j++) {
                list[j] = ToSigned(tempList[j]);
            }
            delete[] tempList;
        }
        else {
            for(int j=0;j<16;j++) {
                list[j] = ToSigned(b.get(8));
            }
        }
        QMatrix4x4 chunk = listToMatrix(list);
        chunk = DeQuant(chunk);
        chunk = this->inverseDCT(chunk);
        // Set numbers into picture matrix
        for(int k=0;k<4;k++) {
            for(int l=0;l<4;l++) {
                image[k+row][l+col] = (int)(chunk(k,l)+128);
            }
        }
        col += 4;
        // our col index is out of bounds, move down 4 rows and start left
        if(col >= set.Width()) {
            col = 0;
            row += 4;
        }

        // Handle next chunk
    }
    // Write image matrix to file
    delete[] list;
    //  WriteToFile(image);
    util::BitStreamWriter b2(size);
    for(int i=0;i<set.Height();++i)
        for(int j=0;j<set.Width();++j) {
        b2.put(8,image[i][j]);
    }
    for(int i=0;i<set.Height();++i)
        free(image[i]);
    free(image);
    return b2;
}

QMatrix4x4 Decoder::listToMatrix(int *list) {
    QMatrix4x4 matrix;
    matrix.setRow(0,QVector4D(list[0],list[1],list[5],list[6]));
    matrix.setRow(1,QVector4D(list[2],list[4],list[7],list[12]));
    matrix.setRow(2,QVector4D(list[3],list[8],list[11],list[13]));
    matrix.setRow(3,QVector4D(list[9],list[10],list[14],list[15]));
    return matrix;
}

QMatrix4x4 Decoder::DeQuant(QMatrix4x4 m) {
    for(int i=0;i<4;++i) {
        for(int j=0;j<4;++j) {
            m(i,j) = m(i,j) * set.Quanti()(i,j);
            m(i,j) = Round(m(i,j));
        }
    }
    return m;
}

void Decoder::WriteToFile(int**image) {
    ofstream out;
    uint8_t* buff = new uint8_t[set.Width()*set.Height()];
    util::BitStreamWriter b(buff,sizeof(buff));
    for(int i=0;i<set.Height();++i)
        for(int j=0;j<set.Width();++j) {
        b.put(8,toUnsigned(image[i][j]));
    }
    out.open(set.DecFile().toLatin1(),ios_base::binary);
    util::write(out,b);
    out.close();
    free(buff);
}


uint8_t* Decoder::DecodeList(util::BitStreamReader &b) {
    uint8_t *list = new uint8_t[16];
    int total=0;
    int index = 0;
    while(total < 16) {
        int num = b.get(8);
        if((num & 128) == 128) {
            num = num - 128;
            uint8_t x = b.get(8);
            for(int i=0;i<=num;++i) {
                list[index] = x;
                total++;
                index++;

            }
        }
        else {
            for(int i=0;i<=num;++i) {
                list[index] = b.get(8);
                total++;
                index++;
            }
        }
    }
    return list;
}

Using Qt matrices and the BitStream class was given to us and is supposed to be working correctly.
The error im getting is

*** glibc detected *** /media/USB DISK/videocoder/videocoder: free(): invalid next size (fast): 0x0912b368 ***

followed by a trace

any help would be greatly appreciated as i'm getting pretty desperate now.

[Richard.J]

In Decoder::ReadFile you are allocating input via new, but later on, you call free on it. That's definitely wrong. You must call delete[].
In Decoder::WriteToFile there is a similar problem with buff.
I didn't check the rest. Carefully review your code.

[GNiewerth]

Instead of using raw C-style arrays for temporary variables you should seriously consider using std::vector instead.
I took a glance at your code and noticed some strange things:

- inverseDCT
1) You pass the input matrix by value, passing it by const reference might be better since you avoid copying a matrix object (what might be expensive)
2) You call set.DCT() twice, I don´t know what it exactly does, but it sounds expensive. Maybe you can rewrite your code as

QMatrix4x4 Decoder::inverseDCT( const QMatrix4x4& y) 
{
    const QMatrix4x4& DCT = set.DCT();
    return DCT.transposed() * y * DCT;
}

- ReadFile
You allocate two buffers, read into the first, copy the data from first to second and return the second. Looks weird...

Didn´t look closer at the rest of the code.