I think it was a case of being to tired to see the error in my ways. I am using ninject for DI, and I wasn't adding my custom adapter to to the FluentValidationModelValiadtorProvider properly, hence why the client side rule where never getting added.
This is why you shouldn't code when your on 2 hours of sleep :)
And just in case someone else needs help, you can take a look at this stackoverflow question which has a nice walkthrough of whats needed. I wish I found this before falling through this :)