Android : Aligning a RecyclerView below a LinearLayout

I have a linear layout called "footer" I want to place below my RecyclerView(a new type of list introduced in API 21).

1. 
   

   The recyclerview will probably want to have a "match_parent" height to efficiently use memory(avoid unecessary calculation).

   
   


2. 
   

   I shouldnt have the recyclerview first because then it will take up the whole screen and the footer wont be visible.

   
   


3. 
   

   Since the recyclerview comes second the linearlayout shouldnt have a "match_parent" height because then the recyclerview wont be seen, and I don't need the footer to stretch anyways.

   
   

So I have the linearlayout "footer" have a "wrap_content" height first in my root relativelayout, then the recyclerview be "match_parent", and specify it above the "footer":

<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
android:layout_width="match_parent"
android:layout_height="match_parent"
android:paddingBottom="@dimen/activity_vertical_margin"
android:paddingLeft="@dimen/activity_horizontal_margin"
android:paddingRight="@dimen/activity_horizontal_margin"
android:paddingTop="@dimen/activity_vertical_margin">

<LinearLayout
    android:id="@+id/footer"
    android:layout_width="match_parent"
    android:layout_height="wrap_content"
    android:orientation="vertical">

    <ImageView
        android:id="@+id/addTaskImage"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_gravity="end"
        android:src="@drawable/ic_edit"
        android:visibility="visible" />

    <RelativeLayout
        android:id="@+id/addTaskRL"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:visibility="gone">

        <EditText
            android:id="@+id/taskET"
            android:layout_width="match_parent"
            android:layout_height="150sp"
            android:background="@color/White">
        </EditText>

        <RelativeLayout
            android:id="@+id/paletteRed"
            android:layout_width="30dp"
            android:layout_height="30dp"
            android:layout_below="@id/taskET"
            android:layout_marginLeft="50dp"
            android:layout_marginTop="10dp"
            android:background="@color/Red" />

        <RelativeLayout
            android:id="@+id/paletteOrange"
            android:layout_width="30dp"
            android:layout_height="30dp"
            android:layout_below="@id/taskET"
            android:layout_marginLeft="10dp"
            android:layout_marginTop="10dp"
            android:layout_toRightOf="@id/paletteRed"
            android:background="@color/Orange" />

        <RelativeLayout
            android:id="@+id/paletteGreen"
            android:layout_width="30dp"
            android:layout_height="30dp"
            android:layout_below="@id/taskET"
            android:layout_marginLeft="10dp"
            android:layout_marginTop="10dp"
            android:layout_toRightOf="@id/paletteOrange"
            android:background="@color/Green" />

        <RelativeLayout
            android:id="@+id/paletteRoyalBlue"
            android:layout_width="30dp"
            android:layout_height="30dp"
            android:layout_below="@id/taskET"
            android:layout_marginLeft="10dp"
            android:layout_marginTop="10dp"
            android:layout_toRightOf="@id/paletteGreen"
            android:background="@color/RoyalBlue" />

        <RelativeLayout
            android:id="@+id/palettePurple"
            android:layout_width="30dp"
            android:layout_height="30dp"
            android:layout_below="@id/taskET"
            android:layout_marginLeft="10dp"
            android:layout_marginTop="10dp"
            android:layout_toRightOf="@id/paletteRoyalBlue"
            android:background="@color/Purple" />

        <Button
            android:id="@+id/submitBtn"
            android:layout_width="70dp"
            android:layout_height="30dp"
            android:layout_below="@id/taskET"
            android:layout_marginLeft="10dp"
            android:layout_marginTop="10dp"
            android:layout_toRightOf="@id/palettePurple"
            android:background="@color/SkyBlue"
            android:textColor="@color/White"
            android:text="@string/submit">
        </Button>
    </RelativeLayout>
</LinearLayout>

<android.support.v7.widget.RecyclerView
    android:id="@+id/tasksRV"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:layout_above="@id/footer"/>
</RelativeLayout>

I know this is a layout issue only because if I remove the footer my recyclerview is displayed. What is my misunderstanding of layouts above? Thanks