Grrr. This was super aggravating.
The issue that we were trying to solve was this:
We had a SOP Order for a quantity of 10
We sent through an eConnect update to that SOP line, updating the QTYTBAOR = 7, QTYFULFIL = 7 and the QUANTITY = 3 (We're backordering 3, invoicing 7)
Then we transferred the Order to Invoice.
After transferring to invoice, the Order had this

This issue now is how to get the remaining three invoiced?
Enter our two errors mentioned above. I did this about 40-50 times, couldn't get it. In the end, the answer was this:
Set
TOTALQTY = 10
QUANTITY = 3
QTYFULFI = 3
And, use this for the XTNDPRCE:
If .TOTALQTY > 0 Then
.XTNDPRCE = (.UNITPRCE - .MRKDNAMT) * .TOTALQTY
Else
.XTNDPRCE = (.UNITPRCE - .MRKDNAMT) * (.QUANTITY + .QTYTBAOR + .QTYCANCE)
End If
Much thanks to Nate Haaland at Microsoft Support for suggesting the answer.