In my opinion, the biggest drawback with using a speaker level converter is the sound quality suffers. You are taking a signal that is amplified by the head unit, and stepping it back down to pre-amp levels. You are always going to get some noise when you amplify a signal, especially from a factory head unit. Then you reduce the signal level (with noise from added the head unit), pipe that into another amplifier, the output of which adds more noise (albeit usually much less than the factory head unit). So you have the noise added from the head unit combined with whatever noise you get from your amplifier, and the sound quality is much poorer than you would have with an aftermarket head unit with pre-amp outputs going into an amp.
Sorry about the book I just wrote, but I wanted to make it clear to anyone who is planning on doing this. You will definitely get better sound quality with aftermarket.