Android : Android multitouch View.OnTouchListener

Im trying to play around with some multitouch in Android.

I have two RelativeLayout on screen where I would like to get touch events.

fooLayout = (RelativeLayout)findViewById(R.id.foo);
fooLayout.setOnTouchListener(onTouchListener);

barLayout = (RelativeLayout)findViewById(R.id.bar);
barLayout.setOnTouchListener(onTouchListener);

...

View.OnTouchListener onTouchListener = new View.OnTouchListener() {
    @Override
    public boolean onTouch(View v, MotionEvent event) {
    ...

I tried having a OnTouchListener for each layout but that didn't seem to work at all.

I figured since the onTouch method gets an View I would be able to make out with view the current touch event came from and save away the pointer id.

int pointerId = event.getPointerId(event.getActionIndex());
int action = event.getActionMasked();
switch (action) {
     case MotionEvent.ACTION_DOWN:
     case MotionEvent.ACTION_POINTER_DOWN:
          switch (v.getId()) {
               case R.id.foo:
                    fooPointerId = pointerId;
                    break;
               case R.id.bar:
                    barPointerId = pointerId;
                    break;
}

This does not work as I thought it would. When my first finger touches in one of the layouts I will get the correct id from that layout. But when I put down a second finger in the other layout it will get the id of the layout where I put down my first finger.

Am I going about this the wrong way?

Is this possible to do or do I have to put the OnTouchListener on the top view and figure out which layout Im touching by the x/y values of the event?