Each workout creates an inroad into the muscles strength. Working out before your muscle is fully recovered, while not detrimental, will only further deepen the inroad and your strength gains will most likely stall. It's like starting at 100 units of strength and after a workout you're at 70 units. You could workout at 80 or 90 units since you've recovered a bit but you'd rather be at 100 again, and if you wait long enough you might reach 110 units of strength (an increase in the weight or TUL that you use). Dr. Doug McGuff explains this inroad process really well in his book "Body by Science".
When trying to increase strength, it's always best to err on the side of waiting longer. Rest and sleep are what help muscles recover and grow, you won't lose strength for taking an extra day to rest.