Rather than - shows preference. This expression is generally used in 'parallel' structures. e.g - with two nouns, adjectives, adverbs, infinitives or -ing forms.
e.g.
1). We ought to invest in machinery rather than buildings.
2). I prefer starting early rather than leaving things to the last minute.
When the main clause has a to - infinitive, rather than is usually followed by an infinitive without to or -ing form.
e.g. - I decided to write rather than phone/phoning.
Instead of - suggests that one person, thing or action replaces another. Instead is not used alone as a preposition; we use the two words instead of.
Instead of is not usually followed by an infinitive.
e.g.
1). I'll have tea instead of coffee, please.
2). I stayed in bed all day instead of going to work.
3). Amit was invited to the reception, but he was ill, so Akash went instead of him.
Note - Instead (without of) is an adverb. It begins or ends a clause usually.
e.g. - She didn't go to Greece after all. Instead , she went to America.
NOTE:
Usage --- instead of + noun phrase. Instead of is only a preposition and can introduce only a phrase i.e no verb.
Usage --- rather than + verb (or) rather than + noun. Further rather than can act as a preposition and can introduce a prepositional phrase or can act as a conjunction and introduce a clause