That’s not a problem.  You can cancel your rider at any time without impacting your life insurance policy.