Here are some tests I did for one character. The number on the left indicates the new value of Agility, then to the right: time it struck this value, and how much time it elapsed before raising it (running around the world map) or abusing it (resting in New Attnam). There are some snack breaks I didn't count.
12 - 1/12:00
13 - 1/12:40 - 0:40 to raise from 12 to 13
12 - 1/22:05 - 9:25 to drop from 13 to 12
13 - 1/22:35 - 0:30 to raise from 12 to 13
12 - 2/9:35 - 11:00 to drop from 13 to 12
13 - 2/4:50 - 0:15 to raise from 12 to 13
12 - 2/11:20 - 6:30 to drop from 13 to 12
12 - 2/16:20 - 5:00 extra rest
13 - 2/16:55 - 0:35 to raise from 12 to 13
14 - 2/19:05 - 2:10 to raise from 13 to 14
15 - 3/4:05 - 9:00 to raise from 14 to 15
16 - 3/6:50 - 2:45 to raise from 15 to 16
17 - 3/10:15 - 3:25 to raise from 16 to 17
16 - 3/12:20 - 3:05 to drop from 17 to 16
17 - 3/12:45 - 0:25 to raise from 16 to 17
18 - 3/16:45 - 4:00 to raise from 17 to 18
19 - 3/21:35 - 4:50 to raise from 18 to 19
20 - 4/5:00 - 8:25 to raise from 19 to 20
19 - 4/7:55 - 2:55 to drop from 20 to 19
20 - 4/9:35 - 1:40 to raise from 19 to 20
21 - 4/22:45 - 13:10 to raise from 20 to 21
20 - 5/1:45 - 3:00 to drop from 21 to 20
19 - 5/13:30 -11:45 to drop from 20 to 19
18 - 6/1:00 -11:30 to drop from 19 to 18
It seems that:
1. Once a stat drops, the counter for raising it is reset, and so instantly doing something stat-raising bumps the stat back to its previous value, and then it takes the regular amount of time to raise that stat to the next level.
(I was secretly hoping that the time-to-raise value would get reset and so it would let you raise stats up quickly if you did it immediately after they dropped, but no such luck apart from getting them in their previous place)
2. The time to drop a stat does not depend whether it was raised a long or a short time ago, instead, it depends on the height of the stat (the higher it is, the faster it drops).
There are some weird anomalies: unless I got something wrong, after dropping from 13 it 12 Agility, it took me just over 30 minutes to get it back up, then over 2 hours to raise it further to 14, then NINE hours to raise it to 15, and then less than 3 hours to raise it to 16 and over 3 hours to raise it to 17. Where the hell did those 9 hours come from?
(it seemed to me so far that stat raising was regular... chances are, there may be something like adding a random amount of points to stat increase/decrease every turn in addition to preset values, or some other weird factors).
Gah, gotta check the source files again