The "of" in "instead of" is a preposition, and its object must be a noun. Thus, "instead of" can only be used to compare single nouns, and the GMAT Sentence Correction seems to avoid even this use. The GMAT also seems to avoid the "instead of" [gerund] structure. The "than" in "rather than" is a subordinate clause, so the structure "rather than" is considerably more flexible in what can follow it.