Cyclic orbit codes are a prominent class of subspace codes, generated by taking the orbit of a single subspace of the finite field $F_{q^n}$ under an action of a Singer subgroup. We are interested in classifying the isometry classes of these codes for various parameters. In order to do this, we show that the automorphism group of a cyclic orbit code is heavily related to the smallest subfield of the ambient field which contains a generating subspace for the code. When there is no proper subgroup of the ambient field containing a generator for the code, we see that any possible isometry is an element of the normalizer of the Singer subgroup.