Maximum number of connections with fixed_total_number

[AlexVanMeegen]

Describe the bug
With fixed_total_number, it is not possible to create more than 2^32 - 1 synapses in a single connect call independent of the number of virtual processes. A BadParameterValue exception is thrown.

To Reproduce
Run this minimal example in any parallelization setup of your liking with a recent NEST version (probably after after #1299): max_fixed_total_number.zip

Additional context
It seems to me that the problem arises when the connections are assigned to the virtual processes using a binomial distribution, i.e. here in the current master. In the first run of the loop, all synapses have to be distributed and thus N equals the total number of synapses on all virtual processes.

Hat tip and big thanks to @jarsi for helping me to get this far.

[suku248]

@hakonsbm could you please take a look at this?

[hakonsbm]

When using fixed_total_number, NEST draws from a binomial distribution, iterating over all VPs until the specified number of connections is created. If NEST is installed with GSL, gsl_ran_binomial() is used, where the number of trials is given as an unsigned int.

With an unsigned int width of 32 bits, the maximum number of trials possible is 2^32 - 1, and the exception raised,

BadParameterValue in Connect_g_g_D_D: gsl_binomial RDV: N < 4.29497e+09 required.

is when the check that the specified value is within this limit fails, here:

nest-simulator/librandom/gsl_binomial_randomdev.cpp

Lines 95 to 103 in d713087

	librandom::GSL_BinomialRandomDev::set_n( size_t n_s )
	{
	// gsl_ran_binomial() takes n as an unsigned int, so it cannot be greater
	// than what an unsigned int can hold.
	const auto N_MAX = std::numeric_limits< unsigned int >::max();
	if ( n_s >= N_MAX )
	{
	throw BadParameterValue( String::compose( "gsl_binomial RDV: N < %1 required.", static_cast< double >( N_MAX ) ) );
	}

It is therefore not currently possible to create more than 2^32 - 1 synapses with fixed_total_number in a single Connect call. However, the error message could be more informative.

Note that with the new random generator setup developed for NEST 3.0 (#1549), number of trials for the binomial distribution is given as an unsigned long, which on my machine has a width of 64 bits, and a maximum limit of 2^64 - 1.

[AlexVanMeegen]

Thanks for the more detailed explanation, @hakonsbm. This is precisely what I tried to express in the issue. Once #1549 is merged, the increase to 2^64 - 1 would fix it for all practical purposes I guess. ^^

Still, I think that this is an undesired behavior - why should there by an upper limit on the number of connections? In particular, I don't understand why the number of connection have to be binomially distributed across the VPs and not simply give each VP the same number of connections (which would immediately fix this issue as far as I can see). Clearly, that's entirely due to me not being familiar with these NEST internals. Could you elaborate on this just for my information (or point me to the relevant reference)?

[hakonsbm]

In particular, I don't understand why the number of connection have to be binomially distributed across the VPs and not simply give each VP the same number of connections (which would immediately fix this issue as far as I can see).

I'm not sure why it's done this way. Maybe @heplesser or @jougs knows?

[heplesser]

@AlexVanMeegen Strictly speaking, it is not a binomal, but a multinomial distribution. If we assigned a fixed number of connections to each VP, we would introduce a regularity on the connection pattern that should not be there. For the most extreme case, consider that the number of VPs is equal to the number of connections to be created: Then we would have exactly on connection per VP, which certainly is too regular. You may find a more detailed discussion in the original Potjans & Diesmann (2014) paper.

Note also that 2^64 ~ 10^19, while there are only an estimated 10^15 synapses in the human brain—so we should be safe ;).

[AlexVanMeegen]

Thanks for the explanations and the reference, @heplesser and @hakonsbm!