NAG Library Chapter Introduction

G05 – Random Number Generators

A sequence of pseudorandom numbers is a sequence of numbers generated in some systematic way such that they are independent and statistically indistinguishable from a truly random sequence. A pseudorandom number generator (PRNG) is a mathematical algorithm that, given an initial state, produces a sequence of pseudorandom numbers. A PRNG has several advantages over a true random number generator in that the generated sequence is repeatable, has known mathematical properties and can be implemented without needing any specialist hardware. Many books on statistics and computer science have good introductions to PRNGs, for example Knuth (1981) or Banks (1998).

PRNGs can be split into base generators, and distributional generators. Within the context of this document a base generator is defined as a PRNG that produces a sequence (or stream) of variates (or values) uniformly distributed over the interval 0,1. Depending on the algorithm being considered, this interval may be open, closed or half-closed. A distribution generator is a routine that takes variates generated from a base generator and transforms them into variates from a specified distribution, for example a uniform, Gaussian (Normal) or gamma distribution.

The period (or cycle length) of a base generator is defined as the maximum number of values that can be generated before the sequence starts to repeat. The initial state of the base generator is often called the seed.

There are six base generators currently available in the NAG Library, these are; a basic linear congruential generator (LCG) (referred to as the NAG basic generator) (see Knuth (1981)), two sets of Wichmann–Hill generators (see Maclaren (1989) and Wichmann and Hill (2006)), the Mersenne Twister (see Matsumoto and Nishimura (1998)), the ACORN generator (see Wikramaratna (1989)) and L'Ecuyer generator (see L'Ecuyer and Simard (2002)).

The performance of the NAG basic generator has been analysed by the Spectral Test, see Section 3.3.4 of Knuth (1981), yielding the following results in the notation of Knuth (1981).

The right-hand column gives an upper bound for the values of νn attainable by any multiplicative congruential generator working modulo 259.

An informal interpretation of the quantities νn is that consecutive n-tuples are statistically uncorrelated to an accuracy of 1/νn. This is a theoretical result; in practice the degree of randomness is usually much greater than the above figures might support. More details are given in Knuth (1981), and in the references cited therein.

Note that the achievable accuracy drops rapidly as the number of dimensions increases. This is a property of all multiplicative congruential generators and is the reason why very long periods are needed even for samples of only a few random numbers.

This series of Wichmann–Hill base generators (see Maclaren (1989)) use a combination of four linear congruential generators and has the form: where the ui, for i=1,2,…, form the required sequence. The NAG Library implementation includes 273 sets of parameters, aj,mj, for j=1,2,3,4, to choose from.

The constants ai are in the range 112 to 127 and the constants mj are prime numbers in the range 16718909 to 16776971, which are close to 224=16777216. These constants have been chosen so that each of the resulting 273 generators are essentially independent, all calculations can be carried out in 32-bit integer arithmetic and the generators give good results with the spectral test, see Knuth (1981) and Maclaren (1989). The period of each of these generators would be at least 292 if it were not for common factors between m1-1, m2-1, m3-1 and m4-1. However, each generator should still have a period of at least 280. Further discussion of the properties of these generators is given in Maclaren (1989).

This Wichmann–Hill base generator (see Wichmann and Hill (2006)) is of the same form as that described in Section 2.1.2, i.e., a combination of four linear congruential generators. In this case a1=11600, m1=2147483579, a2=47003, m2=2147483543, a3=23000, m3=2147483423, a4=33000, m4=2147483123.

Unlike in the original Wichmann–Hill generator, these values are too large to carry out the calculations detailed in (1) using 32-bit integer arithmetic, however, if then setting gives and Wi can be calculated in 32-bit integer arithmetic. Similar expressions exist for xi, yi and zi. The period of this generator is approximately 2121.

Further details of implementing this algorithm and its properties are given in Wichmann and Hill (2006). This paper also gives some useful guidelines on testing PRNGs.

The Mersenne Twister (see Matsumoto and Nishimura (1998)) is a twisted generalized feedback shift register generator. The algorithm underlying the Mersenne Twister is as follows:

(i)	Set some arbitrary initial values x1,x2,…,xr, each consisting of w bits.
(ii)	Letting A= 0 Iw-1 aw aw-1⋯a1 , where Iw-1 is the w-1×w-1 identity matrix and each of the ai,i=1 to w take a value of either 0 or 1 (i.e., they can be represented as bits). Define x i+r = x i+s ⊕ x i ω : l+1 | x i+1 l:1 A , where x i ω : l+1 | x i+1 l:1  indicates the concatenation of the most significant (upper) w-l bits of xi and the least significant (lower) l bits of xi+1.
(iii)	Perform the following operations sequentially: z = xi+r ⊕ xi+r ≫ t1 z = z ⊕ z ≪ t2 ​ AND ​ m1 z = z ⊕ z ≪ t3 ​ AND ​ m2 z = z ⊕ z ≫ t4 u i+r = z/ 2w - 1 , where t1, t2, t3 and t4 are integers and m1 and m2 are bit-masks and ‘≫t’ and ‘≪t’ represent a t bit shift right and left respectively, ⊕ is bit-wise exclusively or (xor) operation and ‘AND’ is a bit-wise and operation.

The ui+r, for i=1,2,…, form the required sequence. The supplied implementation of the Mersenne Twister uses the following values for the algorithmic constants: where the notation 0xDD … indicates the bit pattern of the integer whose hexadecimal representation is DD ….

This algorithm has a period length of approximately 219,937-1 and has been shown to be uniformly distributed in 623 dimensions (see Matsumoto and Nishimura (1998)).

The ACORN generator is a special case of a multiple recursive generator (see Wikramaratna (1989) and Wikramaratna (2007)). The algorithm underlying ACORN is as follows:

(i)	Choose an integer value k≥1.
(ii)	Choose an integer value M, and an integer seed Y00, such that 0<Y00<M and Y00 and M are relatively prime.
(iii)	Choose an arbitrary set of k initial integer values, Y01,Y02,…,Y0k, such that 0≤ Y0m<M, for all m=1,2,…,k.
(iv)	Perform the following sequentially: Y i m = Y i m-1 + Y i-1 m  mod  M for m=1,2,…,k.
(v)	Set ui=Yik/M.

The ui, for i=1,2,…, then form a pseudorandom sequence, with ui ∈0,1, for all i.

Although you can choose any value for k, M, Y00 and the Y0m, within the constraints mentioned in (i) to (iii) above, it is recommended that k≥10, M is chosen to be a large power of two with M≥260 and Y00 is chosen to be odd.

The period of the ACORN generator, with the modulus M equal to a power of two, and an odd value for Y00 has been shown to be an integer multiple of M (see Wikramaratna (1992)). Therefore, increasing M will give a series with a longer period.

The base generator L'Ecuyer MRG32k3a (see L'Ecuyer and Simard (2002)) combines two multiple recursive generators: where a11 = 0 , a12 = 1403580 , a13 = -810728 , m1 = 232-209 , a21 = 527612 , a22 = 0 , a23 = -1370589 , m2 = 232-22853 , and ui , i = 1 , 2 ,…  form the required sequence. If d=m1 then ui∈0,1 else if d=m1+1 then ui∈0,1. Combining the two multiple recursive generators (MRG) results in sequences with better statistical properties in high dimensions and longer periods compared with those generated from a single MRG. The combined generator described above has a period length of approximately 2191.

Three types of low-discrepancy sequences are supplied in this library, these are due to Sobol, Faure and Niederreiter. Two sets of Sobol sequences are supplied, the first is based on work of Joe and Kuo (2008) and the second on the work of Bratley and Fox (1988). More information on quasi-random number generation and the Sobol, Faure and Niederreiter sequences in particular can be found in Bratley and Fox (1988) and Fox (1986).

This implementation of scrambled quasi-random sequences is based on TOMS algorithm 823 and details can be found in the accompanying paper, Hong and Hickernell (2003). Three methods of scrambling are supplied; the first a restricted form of Owen's scrambling (Owen (1995)), the second based on the method of Faure and Tezuka (2000) and the last method combines the first two.

Scrambled versions of both Sobol sequences and the Niederreiter sequence can be obtained.

The efficiency of a simulation exercise may often be increased by the use of variance reduction methods (see Morgan (1984)). It is also worth considering whether a simulation is the best approach to solving the problem. For example, low-dimensional integrals are usually more efficiently calculated by routines in Chapter D01 rather than by Monte–Carlo integration.

A copula is a function that links the univariate marginal distributions with their multivariate distribution. Sklar's theorem (see Sklar (1973)) states that if f is an m-dimensional distribution function with continuous margins f1 , f2 ,…, fm , then f has a unique copula representation, c, such that

The copula, c, is a multivariate uniform distribution whose dependence structure is defined by the dependence structure of the multivariate distribution f, with where ui ∈ 0,1 . This relationship can be used to simulate variates from distributions defined by the dependence structure of one distribution and each of the marginal distributions given by another. For additional information see Nelsen (1998) or Boye (Unpublished manuscript) and the references therein.

The one exception to this is the Wichmann–Hill II generator. The Wichmann and Hill (2006) paper describes a method of generating blocks of variates, with lengths up to 290, by fixing the first three seed values of the generator (w0, x0 and y0), and setting z0 to a different value for each stream required. This is similar to the skip-ahead method described in Section 2.7.3, in that the full sequence of the Wichmann–Hill II generator is split into a number of different blocks, in this case with a fixed length of 290. But without the computationally intensive initialization usually required for the skip-ahead method.

This method of producing multiple streams can also be used for the Sobol and Niederreiter quasi-random number generator via the parameter ISKIP in G05YLF.

Since calculating Av by a standard square and multiply algorithm is Ok3 log⁡v and requires over 47MB of memory (see Haramoto et al. (2008)), an indirect calculation is performed which relies on a property of the characteristic polynomial pz of A, namely that pA=0. We then define and observe that for a polynomial qz. Since pA=0, we have that g A = A v  and

This polynomial evaluation can be performed using Horner's method: which reduces the problem to advancing the generator k-1 places from state xn and adding (where addition is as defined over 𝔽2) the intermediate states for which ai is non-zero.

There are therefore two stages to skipping the Mersenne Twister ahead v places:

(i)	Calculate the coefficients of the polynomial g z = z v  mod  p z ;
(ii)	advance the sequence k-1 places from the starting state and add the intermediate states that correspond to non-zero coefficients in the polynomial calculated in the first step.

The resulting state is that for position v in the sequence.

The cost of calculating the polynomial is O k 2 log⁡v  and the cost of applying it to state is constant. Skip ahead functionality is typically used in order to generate n independent pseudorandom number streams (e.g., for separate threads of computation). There are two options for generating the n states:

(i)	On the master thread calculate the polynomial for a skip ahead distance of v and apply this polynomial to state n times, after each iteration j saving the current state for later usage by thread j.
(ii)	Have each thread j independently and in parallel with other threads calculate the polynomial for a distance of j+1⁢v and apply to the original state.

Since lim v → ∞ log⁡v = log⁡ n ⁢ v , then for large v the cost of generating the polynomial for a skip ahead distance of n⁢v (i.e., the calculation performed by thread n-1 in option (ii) above) is approximately the same as generating that for a distance of v (i.e., the calculation performed by thread 0). However, only one application to state need be made per thread, and if n is sufficiently large the cost of applying the polynomial to state becomes the dominant cost in option (i), in which case it is desirable to use option (ii). Tests have shown that as a guideline it becomes worthwhile to switch from option (i) to option (ii) for approximately n>30.

Leap frog calculations with the Mersenne Twister are performed by computing the sequence fully up to the required size and discarding the redundant numbers for a given stream.

Prior to generating any pseudorandom variates the base generator being used must be initialized. Once initialized, a distributional generator can be called to obtain the variates required. No interfaces have been supplied for direct access to the base generators. If a sequence of random variates from a uniform distribution on the open interval 0,1, is required, then the uniform distribution routine (G05SAF) should be called.

Prior to generating any variates the base generator must be initialized. Two utility routines are provided for this, G05KFF and G05KGF, both of which allow any of the base generators to be chosen.

G05KFF selects and initializes a base generator to a repeatable (when executed serially) state: two calls of G05KFF with the same argument-values will result in the same subsequent sequences of random numbers (when both generated serially).

G05KGF selects and initializes a base generator to a non-repeatable state in such a way that different calls of G05KGF, either in the same run or different runs of the program, will almost certainly result in different subsequent sequences of random numbers.

No utilities for saving, retrieving or copying the current state of a generator have been provided. All of the information on the current state of a generator (or stream, if multiple streams are being used) is stored in the integer array STATE and as such this array can be treated as any other integer array, allowing for easy copying, restoring, etc.

As mentioned in Section 2.7.1, it is important to note that the statistical properties of pseudorandom numbers are only guaranteed within sequences and not between sequences produced by the same generator. Repeated initialization will thus render the numbers obtained less rather than more independent. In a simple case there should be only one call to G05KFF or G05KGF and this call should be before any call to an actual generation routine.

If a single sequence is required then it is recommended that the Mersenne Twister is used as the base generator (GENID=3). This generator is fast, has an extremely long period and has been shown to perform well on various test suites, see Matsumoto and Nishimura (1998), L'Ecuyer and Simard (2002) and Wichmann and Hill (2006) for example.

When choosing a base generator, the period of the chosen generator should be borne in mind. A good rule of thumb is never to use more numbers than the square root of the period in any one experiment as the statistical properties are impaired. For closely related reasons, breaking numbers down into their bit patterns and using individual bits may also cause trouble.

If the Wichmann–Hill II base generator is being used, and a period of 290 is sufficient, then the method described in Section 2.7.1 can be used. If a different generator is used, or a longer period length is required then generating multiple streams by altering the initial values should be avoided.

Using a different generator works well if less than 277 streams are required.

Of the remaining two methods, both skip-ahead and leap-frogging use the sequence from a single generator, both guarantee that the different sequences will not overlap and both can be scaled to an arbitrary number of streams. Leap-frogging requires no a-priori knowledge about the number of variates being generated, whereas skip-ahead requires you to know (approximately) the maximum number of variates required from each stream. Skip-ahead requires no a-priori information on the number of streams required. In contrast leap-frogging requires you to know the maximum number of streams required, prior to generating the first value. Of these two, if possible, skip-ahead should be used in preference to leap-frogging. Both methods required additional computation compared with generating a single sequence, but for skip-ahead this computation occurs only at initialization. For leap-frogging additional computation is required both at initialization and during the generation of the variates. In addition, as mentioned in Section 2.7.4, using leap-frogging can, in some instances, change the statistical properties of the sequences being generated.

After calling G05RCF or G05RDF the G01F routines in Chapter G01 can be used to convert the uniform marginal distributors into a different form as required.

Prior to generating any quasi-random variates the generator being used must be initialized via G05YLF or G05YNF. Of these, G05YLF can be used to initialize a standard Sobol, Faure or Niederreiter sequence and G05YNF can be used to initialize a scrambled Sobol or Niederreiter sequence.

Due to the random nature of the scrambling, prior to calling the initialization routine G05YNF one of the pseudorandom initialization routines, G05KFF or G05KGF, must be called.

Once a quasi-random generator has been initialized, using either G05YLF or G05YNF, one of three generation routines can be called to generate uniformly distributed sequences (G05YMF), Normally distributed sequences (G05YJF) or sequences with a log-normal distribution (G05YKF). For example, for a repeatable sequence of scrambled quasi-random variates from the Normal distribution, G05KFF must be called first (to initialize a pseudorandom generator), followed by G05YNF (to initialize a scrambled quasi-random generator) and then G05YJF can be called to generate the sequence from the required distribution.

Sequences from other distributions can be obtained by calling the ‘deviate’ routines supplied in Chapter G01 on the results from G05YMF. However, care should be taken when doing this as some of these ‘deviate’ routines are only accurate up to a limited number of significant figures which may effect the statistical properties of the resulting sequence of variates.

For example, if you wish to assign to Z the difference between two successive random numbers generated by G05KAF, beware of writing

Z = G05KAF(IGEN,ISEED) - G05KAF(IGEN,ISEED)

It is quite legitimate for a Fortran compiler to compile zero, one or two calls to G05KAF; if two calls, they may be in either order (if zero or one calls are compiled, Z would be set to zero). A safe method to program this would be

X = G05KAF(IGEN,ISEED)
Y = G05KAF(IGEN,ISEED)
Z = X-Y