Yes, I have that on my Galaxy Nexus as well (generally referred to as "latency" by audio developers). I think it has to do with the size of the audio buffer. From Slokin's comments in SDLActivity.java:
// Let the user pick a larger buffer if they really want -- but ye
// gods they probably shouldn't, the minimums are horrifyingly high
// latency already
desiredFrames = Math.max(desiredFrames, (AudioTrack.getMinBufferSize(sampleRate, channelConfig, audioFormat) + frameSize - 1) / frameSize);
In other words, the problem comes from AudioTrack.getMinBufferSize returning a huge buffer that takes about half a second to fill on the Galaxy Nexus. This was less of a problem on my Droid X. I'll ask on Stack Overflow to see if anyone has a way around this problem.