Net direct tax collection dropped 3.95% to Rs 6.64 lakh crore this fiscal due to higher refunds.