Convert from Base64 to binary string?

[mwpeck]

How can you convert a Base64 string, like AxIRkr80X7KD2tSAwi4SkJA, into a full binary string? I have trying to find something on how to do this but am unable to come up with a solution.....its hard enough finding an online decoder that gives the raw binary string.

[MadHatter]

string b64encodedString = Convert.ToBase64String(new byte[] {31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60});
byte[] buff = Convert.FromBase64String(b64encodedString);
string binary = "";
foreach (byte b in buff) {
    binary += Convert.ToString(b, 2).PadLeft(8, '0') + " ";
}

[mwpeck]

Ok, I see what its doing......its converting the number of the letter to its binary......what is the best way to assign the correct numbers to the letters so I can take the string (like above) and convert that into the binary?

What I am looking to be able to do, is take the sting like above, and convert that into binary, the method you posted works in a way, but I have to manually find the correct corresponding number. For example, A = 0, x = 46, and so on. Any way to have the computer find that or should I build a large if statement that says if 1st character == A, then number = 0 then remove the first character and loop?

Also, once I did something like that, what would be the best way to plug all the numbers into the Convert.ToBase64String statement? It is possible I will not know the length of the base64 string until the program is ran.

[MadHatter]

base 64 is an encoding schema. it allows bytes to be encoded into a sequence of letters, numbers and a couple of other characters. http://en.wikipedia.org/wiki/Base64

what that base64 string represents is a sequence of bits/bytes. all I did, was decode the base 64 string back into its byte array form (byte is the smallest data type you can deal with), then used the convert.tostring method to convert each byte value into a binary sequence: Convert.ToString(byteValue, base) and padded it on the left hand side w/ zero's to maintain widths.

no look up needed is ever needed to convert each character of the base 64 string into binary. that would be taking the ASCII value of each character represented in the base 64 string, and converting that into binary... you're gonna bloat that data set up nice and fat if you do it that way. simply convert the original byte array to a binary string, and you'll save yourself some space.

[mwpeck]

How would I manage that? What I'm trying to do is find the binary equivilent of the Base64 string......the purposes I need it for I actually with 6 places not 8 that typical binary uses, changing the 8 to a 6 in the above code worked fine on doing that. For instance, if I have base64 x which is 0010 1110 in binary, all I need is the 10 1110.



How would I go about converting each character of a base64 string into binary?

PS: Links/tips on how to do it are as good as working code, if not better.

[MadHatter]

first off, let me just say that I think you've confused the hell out of yourself for no reason in particular. my original sample would have sufficed.

base 64 translates an 8 bit value into a 6 bit value using a table of printable characters (ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/ and = for padding).

first they take the binary value of the byte(s) to convert: 255, 254, 253

255 = 11111111
254 = 11111110
253 = 11111101

and they lay them out in order:

11111111 11111110 11111101

then we start taking 6 bits at a time

111111 111111 111011 111101

each of these values are a number from 0 - 63 (ie 6 bits). It just so happens that the base 64 table has 64 characters in it, so each 6 bit value maps to the index of one of the characters in the base 64 table:

111111 = 63 = '/'
111111 = 63 = '/'
111011 = 59 = '7'
111101 = 61 = '9'

so our base 64 encoded version of a 3 byte array { 255, 254, 253} is "//79"

now, when our data being encoded isn't evenly divisible by 6, it has to be padded with zeros. when the data is padded w/ zero's, it has the padding character '=' added to the end of the base 64 character string. this is so the decoder can know how and where to stop decoding bits.

now, whether you go through your base 64 encoded string, and get the index value of each character from the table, then convert that to binary and append it to a string, or whether you decode the base64 encoded string, and append each 8 bit value (as I previously did), you're going to come up with the exact same binary string...

let me show you a proof:

byte[] origBuffer = Encoding.ASCII.GetBytes("hello world");
string base64String = Convert.ToBase64String(origBuffer);
string base64Table = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
string binary = "";

// -- Use the base 64 string to encode a binary string
// ----------------------------------------------------------------
// iterate through each character in our b64 string
// get it's index, and convert that # into binary
// then append it to our binary string:
foreach (char c in base64String) {
    // if we've reached our padding character, we're done
    if (c == '=') break;
    byte idx = (byte)base64Table.IndexOf(c);
    binary += Convert.ToString(idx, 2).PadLeft(6, '0');
}
//
// binary == "011010000110010101101100011011000110111100100000011101110110111101110010011011000110010000"
//

// determine how many 0's are appended to the end of the sequence
int paddedBits = binary.Length % 8;
// remove any leftovers
if (paddedBits > 0) {
    binary = binary.Substring(0, binary.Length - paddedBits);
}

//
// binary == "0110100001100101011011000110110001101111001000000111011101101111011100100110110001100100"
//

// -- Use the original buffer to encode the binary string
// ------------------------------------------------------------------
// If we decode base64String we'll get the origBuffer
// foreach(byte b in Convert.FromBase64String(base64String)) { ... }
string bin2 = "";
foreach (byte b in origBuffer) {
    bin2 += Convert.ToString(b, 2).PadLeft(8, '0');
}

//
//   bin2 == "0110100001100101011011000110110001101111001000000111011101101111011100100110110001100100"
//

bool areEqual = bin2 == binary;
// areEqual is true

numbers are numbers... they can be displayed in various ways, but they always represent the same binary sequence. Whether you convert each 8 bits to a hex character, or whether you encode them using base 64, or utilize an 8 bit sequencing mechanism, you'll always come up w/ the same values.

hope this helps somewhat.

[mwpeck]

Here an oddity I cannot figure out, If I use an base64 string thats already encoded in the first part of your code (with output being the binary variable), it works fine and gives me the correct binary values:

string base64String = "OgNFg4uM25HjWtdaW5GyiPyrDkA";
            string base64Table = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
            string binary = "";

            // -- Use the base 64 string to encode a binary string
            // ----------------------------------------------------------------
            // iterate through each character in our b64 string
            // get it's index, and convert that # into binary
            // then append it to our binary string:
            foreach (char c in base64String)
            {
                // if we've reached our padding character, we're done
                if (c == '=') break;
                byte idx = (byte)base64Table.IndexOf(c);
                binary += Convert.ToString(idx, 2).PadLeft(6, '0') + " ";
            }

It outputs something along the lines of 001110 100000 001101 etc etc, which is the same as if I had done it manually.

However, if I try to do something similar with the second part of your code, it gives me something completely different. For instance:

byte[] origBuffer = Encoding.ASCII.GetBytes("OgNFg4uM25HjWtdaW5GyiPyrDkA");
string bin2 = "";
            foreach (byte b in origBuffer)
            {
                bin2 += Convert.ToString(b, 2).PadLeft(6, '0') + " ";
            }

It outputs 1001111 1100111 1001110 etc etc.

I like the smaller code, but how would I get the output to match up with the correct results of the first part?

[MadHatter]

1. "OgNFg4uM25HjWtdaW5GyiPyrDkA" is not a valid base64 string and will throw a format exception if you try to convert it to a byte array using b64. Did you happen to mean "OgNFg4uM25HjWtdaW5GyiPyrDkA="? the padding char is very important.
2. the first line should use Convert.FromBase64String, not Encoding.ASCII.GetBytes to get origBuffer.
3. unless you plan on stripping spaces from your binary string, don't add them in.
4. if you're using the original byte buffer, pad left 8, not 6. bytes are 8 bits, b64 is 6 bits.

[mwpeck]

Is it possible the program that creates the base64 string doesnt add the padding character?

"OgNFg4uM25HjWtdaW5GyiPyrDkA" is the full string I have, it has no padding characters in it but it is possible the program that creates it knows to add a padding character or something.

Right now, I am adding the spaces just so its easier to read and what not.

I am padding 6 instead of 8 because with base64 only giving a max value of 63, the larges binary I will have is 00111111, the two 0's in front will always be zeroes, so I can leave them off so its easier to work with.

For example, the O comes out to 00001110, I need the far right 6, so I need the 001110, from there I need to reverse every 6 values in the string, so 001110 is now 011100, then I have to read 4 bytes, then reverse those 4 bytes, which gives me 1110, that comes out to 14 in decimal which with the program I am doing this for, a value of 14 tells me the base64 encoded string was created after a specific date.

The full string will be processed the same way (reversing 6 bytes, then taking a pre-defined amount, such as 4 bytes for the first part, then possibly 2 bytes for the second section.....the reversing those bytes, which I will convert into a decimal number that represents a specific thing, such as the 14 from above telling me the string was created before or after a certain date).

EDIT:
Tried your suggestions, I now have:

byte[] origBuffer = Convert.FromBase64String("OgNFg4uM25HjWtdaW5GyiPyrDkA=");
string bin2 = "";
foreach (byte b in origBuffer)
{
      bin2 += Convert.ToString(b, 2).PadLeft(6, '0') + " ";
}

And the output is different, but incorrect from what it should be......the first 6 digits is 111010, they should be 001110.

[MadHatter]

it is not possible (according to the RFC for base 64) to leave the padding character off. What is producing this base64 string? whatever it is, its not using a standards compliant base64 encoding library (not all base64 strings have to be padded, just data sets who's original size is not a multiple of 24)

if you decode the base64 string as I did in my last example (the first round), you pad 6 because you're dealing with the overall bytes 6 bits at a time. I think this is where you start to wander.

when you encode the binary using the bytes, you MUST use all 8 bytes else you will truncate your original data.

take the example I used:

255, 254, 253

is

111111111111111011111101

or

11111111 11111110 11111101

now assume you printed 3 8 bit values, but only used the lower 6 bits you'd loose the top 2 bits of every byte so that instead of taking bits

bits 0-5 as your first value
bits 6-11 as your second value
bits 12-17 as your third value and
bits 18-23 as your fourth value

you'd take bits

bits 2-7 as your first bit
bits 10-15 as your second bit and
bits 18-23 as your third bit

and since you skipped the first 2 bits of each of the 3 8 bits, you've completely lost your fourth set.




when dealing with bits, you CANNOT DROP left hand zeros if you plan on stringing them together. dropping zero's moves each bit's location in the resulting string, which as you can see ends up not working...



you're confused about how to create the binary string, thinking that you only want to deal with 6 bits instead of 8, but the only reason that can be done is when you're dealing w/ a base 64 string, because the base 64 string has been re-divided into 6 bit segments. no bits are lost that way, but if you try to only use 6 bits of an 8 bit value, you're changing the original value. zero's are not unimportant, especially in binary. it holds place value, and whether the 1 or 0 is in the first, second, third or fourth position makes ALL the difference in the world.

[MadHatter]

maybe this will help you see it:

the top row are your 3 byte values represented in decimal form
the second row is your 3 byte value represented in binary form
the third row is how base 64 chops up the value. it uses the new values to map a letter from the base 64 table to the value of those 6 bits.

here is your 3 byte values grouped by color in their 8 bit form:


here is that same 3 byte value, grouped by color into 6 bit segments which is how its done in base64 encoding:


so when you take the 8 bit value (as they are in your byte[]) and try to append their binary value to a string, you'll need all 8 bits.

when you obtain the index value of your 6 bit value, you convert that to binary (only 6 bits wide), and append that to your string and you retain all value.

now whether you stack all 8 bits of an 8 bit number end to end, or whether you stack all 6 bit index values of your base 64 string, then end up being the same binary value. if you leave anything out of either one of those processes and you loose info.

[mwpeck]

I think I am misunderstanding how to properly portray what I need across the forum. Basically I'm trying to make a GW utility to handle skill template files......Theres online utilites that basically do the same thing but I wanna do this to try to learn more C#. Here is a page that talks about the format of these base64 encoded strings: http://wiki.guildwars.com/wiki/Skill_template_format

Heres one of the first lines about it:

The skill template file consists of a single string such as ABJRkncAAAoVAAAAAAAA. This string is Base64 encoded. The following format assumes that the Base64 string has been decoded into a binary array.

Here is a sample on how you would manually decode the string (its a different string then given on the main page):

binary            0 0 1 1 1 0   1 0 0 0 0 0   0 0 1 1 0 1   0 0 0 1 0 1   1 0 0 0 0 0 ..
flip each 6 bits  0 1 1 1 0 0  |  0 0 0 0 0 1  |  1 0 1 1 0 0  |  1 0 1 0 0 0  |  0 0 0 0 0 1|..
read sections    |0 1 1 1|0 0 0 0|0 0|0 1 1 0|1 1 0 0|1 0 1 0|0 0 0 0|0 0 0 1|..
                 | type  | version | pl | prim.  | sec.   | attr.  | attrl. |attr.id |..
flip again       |1 1 1 0|0 0 0 0 |0 0 |0 1 1 0 |0 0 1 1 |0 1 0 1 |0 0 0 0 |1 0 0 0 |..
to decimal       |   14  |    0   | 0  |    6   |    3   |    5   |    0   |    8   |..

type (template type)          = 14 (Skill template)
version (template version)    = 0
pl (profession length)        = 0 (=> 4 bits per profession)
prim. (primary profession)    = 6 (Elementalist)
sec. (secondary profession)   = 3 (Monk)
attr. (number of attributes)  = 5 (so 5 attribute sections will follow)
attrl. (attribute length)     = 0 (=> 4 bits per attribute id)
attr.id (attribute id)        = 8 (Air Magic)

Base64's O is 13, which 13 in binary is 00001110, now, if you look at the above example, only the last 6 bits of the binary are shown (only the 001110 instead of the full 8 bit value).

Anyway, once you have that full string, you flip the bytes in six-bit chunks (so you flip 001110 and it becomes 011100), from there the type of the template file (there are two types, equipment and skill, I'm only dealing with skill at the time) takes up 4 bits, so you read the first four bits, then flip those 4 bits yet again (so 0111, the first four bits is flipped and is now 1110). Now, those bits, 1110, is 14 in decimal. For guild wars, 14 in the template files means its a skill template. So you have the first part of the file read, now you go back and read the next four bits, and flip those (in this example, 0000 flipped is of course 0000), which is the template version (no idea why they have a template version of 0, thats just how it is). You continue to do that until you are finished with the binary string you started with. As you can see from the linked page about the templates, the "sections" of bits your reading is predefined, in some parts its defined by what is decoded earlier in the process. Either way you will always know how many bits make up each seperate section.

Maybe I'm misunderstanding what I want to do with this base64 string, maybe you will be able to understand it a little better and lead me to what I am doing wrong.

[MadHatter]

ah guildwars!

they don't use an rfc compliant encoding scheme. they use base 64 mapping to encode the build template.

from what I can see, you simply decode the base64 string into a binary array, then you can iterate over the string to read in the values from it.

from what I've read this should help you decode the base 64 string into a binary string:

public class GWBase64 {
    string table = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
    public string GetBinaryString(string gwb64) {
        string binary = "";
        foreach (char c in gwb64) {
            byte idx = (byte)table.IndexOf(c);
            string littleEndian = Convert.ToString(idx, 2).PadLeft(6, '0');
            for (int i = 5; i >= 0; i--) {
                binary += littleEndian[i];
            }
        }
        return binary;
    }
}

converting the decimal number to binary produces a little endian binary string, since the template decoding schema iterates over the binary string in big endian you have to convert the LE bits to BE (which is what the internal for loop does).

I fired up gw to get a skill template to test with, but ran out of time (gotta go play w/ the kids) to go through it to test. my character was an e/mo and the first few bit sets seemed to be correct, but the second profession showed up as none instead of monk, so I'm not totally convinced this is right, but you should be able to test it.

[mwpeck]

Ok cool, like I said the first half of what you posted earlier seemd to work for it, I didnt go all the way through the string but it seemed correct.

Thanks, I'll mess around with this a bit.....figures GW didnt follow the standards.