X = Type 3 Aug Bonus
Y = Instant Damage on a DoT
Z = Damage Per Application
n = Damage Applications (spell duration + 1)
P = Percentage Bonus

In most cases, the damage bonus is simply:
X = Z * P * n
(Note that "spell duration + 1" is exactly 1 for instant spells, and the + 1 accounts for the extra tick of damage for standard DoTs, so this formula does not need to be different for standard DoTs or DDs.)

However, if this is a DoT spell with an instant damage component, the formula becomes slightly more complex:
X = (Y + Z * n) / (n + 1) * P * n

In plain English, (Y + Z * n) is the total amount of damage the DoT+DD would inflict, and (n + 1) is the number of applications, and (Y + Z * n) / (n + 1) is the average damage per application.

Type 3 aug damage is spread out over the course of the DoT, not taking duration increasing effects into account. In other words, Duration extension effects do not increase or reduce the amount of bonus that a DoT receives per tick. Type 3 aug damage is added to the base value of the nuke, before critical hits are calculated. The final value of X is rounded to the nearest ones place.

Let me know if you have any further questions.