Safe code can invalidate invariants of unsafe code in block containing perfectly safe code.

Famous example is:

    impl Vec {
       pub unsafe fn set_capacity(&mut self, capacity: usize) {

          self.capacity = capacity; // why is that unsafe? 
       }

    }

What OP possibly did was invalidate his invariants in safe code.