See - that's the thing. You just have to practice. First it will seem really bad because when you are trying to say something you automatically think about it in Polish, then you frantically try to remember the English equivalent to translate it and if it doesn't happen instantaneously you panic and it feels like you're constantly forgetting words that you normally (in other situations) know.
The trick is to get to the point where you can switch your mind set from Polish to English and "think" in English as opposed to just quickly translate everything. So when you think of "4" and you're in English mode you should hear in your head "four" not "cztery."
I know that it is hard to do but not impossible, the more you talk to more naturally some it will become. Remember - nobody expects you to be perfect from the start and don't feel you're rushed. Try some simple topics first and then as you become more comfortable move on to those more complex.