Android : Add SwipeRefreshLayout to RecyclerView when it has a empty header

I have added a headerView to my RecyclerView as described in the answer of this question: Is there an addHeaderView equivalent for RecyclerView?

The header is an empty view which will be hidden under toolBar.

<?xml version="1.0" encoding="utf-8"?>
<View xmlns:android="http://schemas.android.com/apk/res/android"
    android:id="@+id/header"
    android:layout_width="match_parent"
    android:layout_height="?attr/actionBarSize" />

The problem is the place of SwipeRefreshLayout. It will be started from Header which is under Toolbar, so it partially hidden under Toolbar.

 <android.support.v4.widget.SwipeRefreshLayout xmlns:android="http://schemas.android.com/apk/res/android"
        xmlns:tools="http://schemas.android.com/tools"
        android:id="@+id/swipe_refresh_layout"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        tools:context="com.hiq.android.client.fragment.BurgerFragment">

        <android.support.v7.widget.RecyclerView
            android:id="@+id/recyclerView"
            android:layout_width="match_parent"
            android:layout_height="match_parent"
            android:scrollbars="vertical" >

        </android.support.v7.widget.RecyclerView>

    </android.support.v4.widget.SwipeRefreshLayout>

How can I solve this problem?