Android : How to stop GridLayout from pushing things up when a view is removed

So I have the GridLayout (When adding a View I don't mess with params), and whenever I remove a View that is directly inside of the layout, it just removes it and pushing everything forward, so when I try to add another View it appends it, instead of adding it where I wanted.

i.e:

0 0 0 0 0 0 1 0 0

0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0

I want to remove 1, and add a 2. Now it's

0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 2

Is there a way to stop this from happening?