No, without a court order dictating when each parent has custody of the kids, there is nothing that would require you to give the children back to your spouse whenever they demand that you do so.