To change the decimal places in the grid itself add a 'DataFormatString' attribute:
<telerik:GridNumericColumn DataField="unitcost" UniqueName="unitcost" HeaderText="Unit Cost" ReadOnly="false" DataFormatString="{0:n5}" ></telerik:GridNumericColumn>
To change it in the edit form, change it in the code behind:
Protected Sub RadGrid1_ItemCreated(ByVal sender As Object, ByVal e As Telerik.Web.UI.GridItemEventArgs) Handles RadGrid1.ItemCreated
If TypeOf e.Item Is GridEditableItem And e.Item.IsInEditMode Then
Dim item As GridEditableItem = CType(e.Item, GridEditableItem)
Dim UnitCostEditor As RadNumericTextBox = CType(item("unitcost").Controls(0), RadNumericTextBox)
UnitCostEditor.NumberFormat.DecimalDigits = 5
End If