For me the biggest issue is to have to trust the Simple X server to not track you.