I hope you realised by now, that for me Yoga is not about the poses and the achievements, as in a WORK OUT, but more a spiritual development, a road to self discovery, as in a WORK IN.
This post, as my beloved teacher would say, is my humble opinion. I always teach you in class to: take it in - digest it - circulate it to every cell - check what feels true to you, keep that - discard the rest (5 Prana Mudra meditation rings a bell dear students? :-)). So please do that, make your own conscious call on the subject.
When we come to physical fitness how does that tie in? To answer the question in the title - is Yoga enough for physical fitness? - in short: it depends. I believe to be in balance both physically, mentally and emotionally you need to know if you are out of balance. If you identified that you have imbalances, the only way back to balance is if you know where you are going, as in what is your balance where you want to be? I wrote a blog post on this according to Ayurveda.
If your imbalance is Vata (excess air and ether elements), and you practice Vata balancing Yoga than you are on the way towards balance. If your imbalance is Kapha (excess earth and water elements) and you practice Vata balancing Yoga than it will take you quite a long while to get back into balance, or you may not even get there. My point is, when you personalise your practice (as it really should be) than yes it could be enough.
On a personal note, I am a Pitta Dosha (fire and water) constitution and had a Kapha imbalance (excess earth and water elements, manifesting as weight gain, sluggishness, accumulating things, not willing to change). I kept on practising Vata balancing Yoga and walking - low impact exercise. Did I get closer to my balance? Honestly? Not really. After I added more Kapha balancing, as in more moderate and high intensity exercises my journey to balance was much quicker. Could I have done it just with Yoga? Yes, practicing Ashtanga Vinyasa Flow or Power Yoga but I preferred to balance my practice with jogging/running in the fresh air.
To summarise my rumblings...I believe Yoga is = union / balance. For me that means a balance of low intensity (fat burning), moderate intensity (cardio) and high intensity (peak) exercise and last but not least, meditation. For you it may be something totally different. My point is, find what works for you, using the very helpful starting points of your Ayurvedic constitution and current imbalance.