Resistances (or "resists") allow a character to reduce a fraction of damage taken based on the type of damage it is. For example, fire resistance decreases the damage taken from any kind of fire damage -- a fire drake's breath, or an archmage's Flame talent, etc. The formula for this is simply damage * (1 - resist%). For instance, 100 damage against 30% resist is 100 * (1 - .3) = 70 damage.
Negative resists occur naturally on some enemy types and can also occur as the result of some artifacts or egos. This causes the target to take more than 100% damage from a hit, using the same formula as usual resistance. For instance, 100 damage against -30% resist is 100 * (1 - -.3) = 130 damage.
"Resist all" is applied independently of single elemental resists. If you have 15% resist all and 20% fire resistance, you will resist 1 - ((1 - 0.15) * (1 - 0.20)) or 32% of all fire damage (in the absence of any fire resistance penetration on your opponent). Resists as displayed on your character sheet include this step.
Resists also have a cap. The standard resistance cap for the player is 70%, which is applied after the 1-(1-a)*(1-b) step shown above. This means that if you have 60% fire resistance and 30% resist all, your combined total resistance would be calculated as 1-(1-0.60)*(1-0.30) = 72%, but then the resistance cap would lower this to 70%. Some equipment or talents may raise the resistance cap for specific elements. In the previous example, if the player wore armor that gave +5% fire resistance cap, then the full 72% would take effect, as that is under the new fire resistance cap of 75%.
Resistance penetration decreases the effective level of your opponent's corresponding resistance multiplicatively. For instance, if your opponent has 30% fire resistance, but you have 10% fire resistance penetration, and you hit your opponent with fire, his effective fire resistance will be reduced to 27%. Resistance penetration cannot cause enemies to take more than 100% damage from a hit. Resistance Penetration is normally capped at 70%.
Resistance Reduction reduces the effective level of your opponent's corresponding resistance additively. For instance, if your opponent has 30% fire resistance and you use a talent that reduces their fire resistance by 20%, they would have 10% fire resistance while under the effect of this talent. However, this can cause enemies to drop into negative resistance. The caveat to this is that it only affects the resistance before all resistance is applied.