Prevent selection change with mouse in a TextBox

Prevent selection change with mouse in a TextBox

In WPF, is there a way to prevent text selection change as a result of
user's mouse activity? I have tried setting IsHitTestVisible to False, but
seems to take away a few other functionalities as well.
I'm basically creating a "masked textbox", that shows TIME in "HH:MM AM"
format and allows user to press Up/Down keys to change the values of Hour,
Minute and AM/PM parts. Left and Right arrow keys let user move between
the parts. The active part is shown through text selection, so I don't
want user to be able to click somewhere to change the selection.