Does 'I was just talking about you' often have a negative connotation?

That expression, by itself, is neutral. The connotation can be determined by the tone and content of what is said next.