ScaledDecimal printString

[Bob Nemec]

Here's an interesting one...

(ScaledDecimal
fromBytes: #[0 0 0 0 100 37 136 0 0 0 0 0 0 0 0 12]
precision: 23
scale: 24) displayString '0.064258799999999998427136'

...will fail with #printString (#displayString is our own method).

(ExCLDTIndexOutOfRange) Index out of range.
In ScaledDecimal>>#printOn:showDigits:pad: precision and scale start off as 23 and 24. These are reduced as trailing zeros are trimmed, until p=6 and s=7, and digits=#(6 4 2 5 8 8 ), at which point it sends...

(p - s + 1) to: p do: [ :i ... (digits at: i)].
or (0) to: 6 do: [:i ... (digits at: i)] ... which fails because i - 0

No really a problem with us, because we use #displayString for all cases where the users sees the number. It's just a bit weird when debugging to see 'Could not print receiver'.

This is on version 7.0

Bob Nemec

[wembley]

Bob -

Here is the comment from ScaledDecimal class>>fromBytes: aByteArray precision: aPrecision scale: aScale

Code: Select all

   "Answer an instance of a ScaledDecimal from the bytes contained in aByteArray,
   with precision aPrecision and scale aScale.

   Requires:
      aByteArray size ==  16.
      aPrecision between: 0 and: PdMaximumPrecision.
      aScale between: 0 and: aPrecision - 1."


So your creation of the ScaledDecimal does not meet the constraints.

Te following code works just fine:

Code: Select all

(ScaledDecimal
fromBytes: #[0 0 0 0 100 37 136 0 0 0 0 0 0 0 0 12]
precision: 25
scale: 24)


Of course it would be better if the printOn: method were a little more defensive in its references to 'p' and 's'. So, in #printOn:showDigits:pad:, I will change:

Code: Select all

"Pad with a leading zero if necessary."
   (p = s) ifTrue: [
      p := p + 1.
      digits addFirst: 0.
      ].


to:

Code: Select all

   "Pad with leading zeros if necessary."
   [p <= s] whileTrue: [
      p := p + 1.
      digits addFirst: 0].


This is not a complete solution since it can still fail if either scale or precision is out of range. To see this, try:

Code: Select all

(ScaledDecimal
fromBytes: #[0 0 0 0 100 37 136 0 0 0 0 0 0 0 0 12]
precision: 37
scale: 24)


I'm curious about the result you showed for your displayString. Why isn't it '0.0642588'? It looks like you are getting a floating-point operation involved somehow.

[Bob Nemec]

I tripped across this number when checking the output of a calculation. It was included in an array of numbers (so the array also could not be printed). I'll see if I can reproduce the math.

As for our #displayString, we've written it deal with some issues on GemStone as well as VA (same code in both). We normally use an explicit number of decimal places, so in this case we'd use #displayStringTo8, which answers '0.06425880'.

Within our #displayString method, we send 'n := n - (n truncated)', which answers 0.0642588s24 ... then, to extract the digits, I change it to a large integer using 'n := ((n * (10 raisedTo: anInteger)) + 0.5) truncated', where anInteger is the requested number of digits. By default, it's the scale of the ScaledDecimal, 24 in this case and, in the case of #displayStringTo8, 8.

This gives 64258799999999998427136, which is where the extra digits come from. Since it rounds off ok, we can live with it. Buy I may look more at the #printOn: method for ScaledDecimal to clean this up in our VA version.

[wembley]

Bob -

I'm considering just how much validation code to add for ScaledDecimal 'precision' and 'scale'. The rules are as follows:

* 31-digits maximum
* 'precision' must be between 1 and 31
* 'scale' must be between 0 and 'precision' - 1

So:

0 <= scale < precision <= 31

Right now 'precision' and 'scale' can be set separately. This makes it almost impossible to verify the relationship. I see a couple possibilities:

1) Verify that 0 < 'precision' <= 31 and 0 <= 'scale' < 31 -- signal an exception if not. Then adapt references to them to account for 'scale' >= 'precision' (there are one or two places in the code where this is done already -- as in #printOn:showDigits:pad:).

2) Change the setters so there is only a #precision:scale: method that can verify 0 <= 'scale' < 'precision' <= 31 -- signal an exception if not. Then the code that accounts for this relationship not being true can be removed.

I think 1) is a little more friendly, but could potentially result in incorrect results. It is also harder to find the places that need to be changed. 2) conforms to the standard's description of the necessary relationships and identifies the (potential) error at the earliest possible moment.

I am leaning toward 2), but I am open to being convinced to go another direction.

[Bob Nemec]

The root cause of this problem seems to be the our use of GemStone ScaledDecimal instances (it's a financial app). GemStone stores ScaledDecimal numbers as fractions, which grow to ridiculous combinations of numerator and denominator as they get changed by math methods.

There can be issues when they get translated into VA ScaledDecimal instances. In most cases we get an exception, saying that the requested precision was too large.

Here's a real example... (float = 29.5271999999999)
(49675785452819768280341882998419/1682373725000000000000000000000) asScaledDecimal

We avoid this problem by rounding our stored numbers to a predefined number of decimal places. We didn't do that in the case, and got the weird ScaledDecimal data.

Ideally, GemStone would fix this by allowing us to map VA ScaledDecimal to GS DecimalFloat, which appears to be a closer fit to the VA ScaledDecimal class structure.

(49675785452819768280341882998419/1682373725000000000000000000000) asDecimalFloat --> 2.9527199999999862266E1

It's an old request to GS, no luck so far.