My wife and I have been sharing a toothbrush for over 5 years this way. From the Oral-B brand actually, so I would dare say that it's a problem with the model bc ours was around 80 euros at the time of purchase so definitely not an expensive model or anything.
We even swap brush heads more since we brush in the morning and in the evening so in total that's 4 swaps a day for 5 years = well over 6000 swaps without ANY issue. The only thing that's wrong with the toothbrush since about a month or two is that the softer plastic grip is starting to get worn out and "peel off"...