No, it does not matter. Depending on the standard you are utilizing either the circle U and the 0.0 to # offset tolerance to indicate inside or outside (See ASME Y14.5-2018) or an offset phantom line may be utilized to clearly indicate the unilateral tolerance - see ISO 1101 any year or ASME Y14.5-2009.