-
Notifications
You must be signed in to change notification settings - Fork 422
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Immutability of distributions #482
Comments
These reallocation would most likely be optimized away by the compiler. I would guess that you would see a performance losses changing to a type because they would not be stored in line and the compiler can do fewer assumptions. |
Interesting. I tried the following test from the REPL (after enclosing the test code in functions), and it turns out that you might as well be right. The first run showed that mutating the mean of a mutable normal is faster, but then subsequent runs show that reallocating immutable normals is at least as fast, perhaps after LLVM's magic kicks in:
|
Just for the record, I also tried setting the mean to a different value at each iteration, to see if the compiler will remain effective, so the following still worked:
|
julia> @code_llvm set_immutable_pdf(n)
define void @julia_set_immutable_pdf_32458(i64) #0 {
top:
ret void
} In this case, LLVM optimized away the whole function because it didn't actually have any affect on anything. This happens in some of the other functions you try to benchmark. |
One more question. What happens if I use a multivariate instead of a univariate distribution? Won't there be unnecessary repeated allocations/deallocations of the parameters of the distribution? I will try it out this evening, and will post the code here. |
For multivariates you can already update the means and covariances in place since they are just vectors and matrices. |
Good point, you actually solved a long-standing question I had. I had been wondering whether it is legitimate (both conceptually and in julian terms) to change a field of type |
So, I ran the simple simulation I had in mind. My first observation/reminder is that for a multivariate normal is simply impossible to do
The fix is to use a loop, but look now how bad performance becomes even by avoiding reallocating an array:
|
Obviously this gap in performance comes from the fact that the mutable multivariate normal (which is not a normal distribution really, but that does not relate to the problem) allows to simply change the reference stored in μ. In the case of the immutable |
You can just create new This will lead to no good. The sparse matrix library in base has been haunted by performance problems because |
|
@KristofferC, I had tried what you proposed (i.e. @eford, yes, the outer loop was used as a test for measuring performance. I agree with what you said that a reference hides the danger of misusing it by mistake, yet when it is used intentionally and carefully it provides better performance than copying elements one by one (and it is indeed possible to use references in the case of MCMC for instance). Now, here is the solution to achieve the end goal of touching the reference without reallocations or copying of values:
So I can simply keep changing |
Having found the above workaround, I can now close the issue. |
After thinking a bit more about the problem, I realised that the solution I provided above is ok as long as one changes values of I agree with your thoughts @KristofferC about performance all around, it makes sense to retain distributions in Distributions as immutable not to lose performance in user cases different than the ones I describe above. For this reason I will keep the current issue closed. The only alternatives I can think to solve the computational problem I am dealing with is either to simply copy-paste all the types of Distributions to a new repo called |
Do you have any real code that you have benchmarked with? |
Only toy examples @KristofferC, but good point, let me set up a proper Gibbs example involving multivariate distributions to see it in practice rather than speculating via satellite examples of smaller caliber. |
Please don't this. I think there are two cases where you would hit problems and both have better solutions that introducing mutable types. Mutable is a terrible name for that type in this context: the important point about those types isn't their mutability, it's the fact that they have identity -- with mutable types you can distinguish
Making distributions mutable is in many ways equivalent to making them container types, which I think is more clearly bad than it sounds to make them mutable. The two issues I see are:
|
Wouldn't it be an option just to sample the random variables directly into the storage of the variable, e.g. something like julia> d = MvNormal(rand(1000), randn(1000,1000) |> t -> t't);
julia> randn!(d.μ); |
I think the best solution to this would be resolving JuliaLang/julia#11902 |
Yes, @johnmyleswhite, it would be nice to have the option to "turn off" @simonbyrne thanks for the link! I read it, and remain a bit unclear as to what the plan re immutability is. I haven't understood if it will ultimately allow to update immutable type fields in place (if yes, this would solve the Distributions issue associated with Gibbs). @andreasnoack your solution seems to be doing what is needed for updating the parameters y in a conditional distribution p(x|y) (as needed for Gibbs)...! At the same time, it sounds I might have to wait till we see how JuliaLang/julia#11902 is resolved... |
In many real applications, it becomes crucial to be able to tweak a distribution. For example, in various Monte Carlo methods (Gibbs with conditional distributions representing block updates, Metropolis-Hastings proposal), it is needed to retain the same type of distribution and only update its parameters. A very concrete example would be to update the mean
μ
of ad = Normal()
. It would be more reasonable to retain the same distribution and only dod.μ = 1.2
, instead of doingd = Normal(1.2)
. The problem with the second solution is that one needs to reallocate the distribution at each iteration - so imagine 100,000 reallocations of a normal if one has 100,000 MCMC iterations.I thought two workarounds; we can simply turn the distributions in Distributions from
immutable
totype
or, if you feel that this is not a good idea, I could simply define a new package with mutable distributions, sayMutableDistributions.jl
. I would favor the former, in order to avoid populating Julia's package ecosystem with multiple distribution-centric packages, but if you feel that turning distributions to mutable types is a big compromise, I am happy to set up a separate package with mutable distributions.The text was updated successfully, but these errors were encountered: