Type that doesn't use much memory?

[Frobot]

I am making a "3 value" array, not really sure what it is called (varName[x, y, z] = Color.Blue)
Each of these 3D points in the array is used to hold a Color value.
So I used- Color[,,]varName= new Color[width, height, depth];
This took up way too much memory and froze everything up (the height, width, and depth can be 500+)
I'm only using maybe 10 different colors, so I changed it to an int array, then assigned each number 1-10 a different color-
int[,,]varName= new int[width, height, depth];
This helped a ton but still can't get up there to the 500+ area (500 ^ 3 different values)
So what type could I use that takes up the least amount of memory?
And also, I notice after I create the array, and even after I close the program out, everything is running very slow.
How can I clear the array after it gets used?
Would varName = null; do anything?

Edit:
I found a list of types and it looks like either byte or sbyte is my best bet.
Or is there anything better?
I'm using positive numbers, 0-10.