The reason you were advised against it was because the region is notorious one. Especially during the Holi time, when anti-social elements are large in number behind the garb of celebrating the festival.
Having said so, I don't think that you should drop your idea. If I were you, I would be particularly careful about the security arrangements during the trip, even if it would increase the budget of the trip.
It's a great experience and you'll enjoy Holi there. it'll be very good and colorful. I just want to suggest you whatever you say the people there won't leave until they put color on you and the people are nice and that's the culture there during holi
so it's advisable to wear waterproof clothes and protect your skin, hair and also your mobiles from colour water. you can roam near banke-bihari Temple and can roam through the streets on that auspicious day.
you can also go for vrindavan parikrama have a Happy and safe trip girls!!