Armor Penetration is a tricky beast for some, i will try to keep this as simple and minimal as posible so it may become easier to comprehend.
First we have to look at what armor penetration (arp) basically is;
Like the name suggests it does something with the armor of the target your are fighting, it 'penetrates' it. What it comes down to is that with 100% arp (1400) you will ignore 80% of the targets armor with your PHYSICAL damage dealing shots. This all comes down to a major dps increase as these shots are in your rotation AND boosted in the MM talent tree (this is why MM prevails over other specs)
When to gem for arp?
Go for arp when you can get to around 600 arp with gems&gear, certainly when you have a trinket that procs arp. keep stacking arp even when your proc will overcap you on arp, you will do more sustained dps.
If you cant reach at least 600 arp you will have to gem for agility as it will then be a better stat per point.
Keep in mind raid buffs like faerie fire or sunder armor will not effect your arp, the armor penetration those buffs offer are for the remaining 20% so it's allways best to get 100% arp.
Now the part where it becomes 'tricky'
Why tricky? well because it's about something that you can't calculate, movement of the player and overall skill/raid awareness.
If a hunter is moving badly and thus missing auto shots, not able to do steady shots on high movement fights, the benefit of arp goes down due to the lack of the main arp shots (steady and auto shots) If you notice you are not yet at the level where you can keep your rotation going whatever happens, you might be better off gemming agility for the time being. There is no shame in this, it actually shows you are very aware and actively improving your class. Also, if you notice a drop in dps when you start (at least 600arp) gemming for arp, probably the reason why is due to this.
This is a basic runthrough of the stat, a lot of websites offer a detailed explaination of the stat and for the number crunchers amongst us, be sure to check those out.