Show us a pic of your light box so we can get a better idea of what you're working with.
For my backdrops, I prefer inexpensive poster board. You can swap colors easily and it's cheap and easy to replace if damaged.
My light box is a heavy duty cardboard box with windows cut into it on the top and the two sides. I bought some inexpensive white fabric from Joann Fabrics (on sale for $2yd). It's thin, so I can layer it if needed for more muting.
I only shoot outdoors when the sun is out. It *does* limit my shooting, but I don't have to store lights or stands for them, and I ALWAYS get perfect white light.
I shoot at f20 almost always, and let the camera choose shutter speed. ISO is usually at 400, but I shoot each pic in -2/3 stop, normal stop, and +2/3 stop on the sensitivity setting. More often than not, the -2/3 stop with a little "brightening" in Canon's image software gives me the lighting I like. I also "sharpen" at 250 on a 500 scale in the same software.
I took this pic yesterday evening, just before dusk, in the shade of my house. Even with no direct light, you still see reflections in the finish, but the light is very soft, preventing hot spots. This one finished up at 0 stop with a bit of brightening to bring out the colors.
Here's a pic showing a mistake--using a blue background for a blue subject. I should have pulled out the black or grey sheet.