Different results between Monte Carlo and Quick Results

Question

I am currently doing an uncertainty analysis for my product system by using Monte Carlo. To do this I removed all uncertainty from the ecoinvent database and thus when I currently run the Monte Carlo I only get one value with no standard deviation. However, this one value is VERY different from the results I get from Quick results, which does not make sense. Should I not get the same value from quick results and Monte Carlo when there is no uncertainty in my system? It seems I get the same value when using Cut-Off but not when using Consequential database, but I cannot see what should make one work and not the other. I have attached pictures for a random ecoinvent process - one being results from the Monte Carlo and the other is the quick result for the same process.

 

Comments

I just double checked and the allocation method is None on both. I also tried using different allocation methods but the results still differed between quick results and Monte Carlo.

Answer 1

Hi, sorry to hear the frustration - we have recently looked into this, and several things:

- this is so far (in my knowledge) primarily important for the ecoinvent database, the only database where uncertainty values are contained

- we will modify the new ecoinvent release 3.11 so that the amount without uncertainty will be the same as the point estimate for the probability distribution

- ecoinvent contains some funny uncertainty values which are obviously incorrect, much too high; in openLCA, we do not correct these, thus these can influence the results. To illustrate this, here two plots of ecoinvent exchanges uncertainty, one for the geometric standard deviation, and one for the few thousand normally distributed exchanges, the coefficient of variance, standard deviation devided by mean.

If you have a few processes in your system where the standard deviation / mean is 1E+17 or also 1E+9, or the exponent in the geometric mean is 1000, it is not surprising to get funny simulation results. Pictures for upcoming ecoinvent 3.11 cutoff but this is similar in earlier versions.

-> we think to add an option in openLCA to clean up obviously nonsense distribution values, but want to make users aware of this and not "sugarcoat" and clean up these just while doing the simulation. That is a difference to Brightway then where this clean up is always applied, for the log normal distribution.

Comments

Thank you for giving this issue attention.

Good idea to automatically adjust the default value to the distribution mean/mode etc. That is a reassuring setting.

I am working with ecoinvent. I am not sure if my results are affected by extreme background SD values.

The uncertainty I want to test is very simple, I only have two variable inputs, in a sub-process of my model. One is a normal distribution, the other is a triangular distr. Both are non-skewed. The default values equal the distribution mean/mode.

When I test the sub-model only, I get the same result from a quick calculation as from a MC run. Very good!

When I test a part of my full model, that is INdependent of the variables I mentioned, I get MC mean that is significantly different than the quick calc. mean.

When I test a part of my full model, that is dependent of the variables I mentioned, I get MC mean that is significantly different than the quick calc. mean.

This indicates two problems:
1) the MC mean is wrong in the sense that it should be very near to the quick calc. mean.
2) I desire to test the resulting uncertainty of my full model, based on only the variables that I focus on (the two I mentioned). I am suspecting that uncertainty from all background processes affects my MC result (i.e., also upstream, ecoinvent processes that I have no 'control over', nor am interested in including the uncertainty of).

I will try to upgrade to a newer version to see if it resolves problem 1.

Can you help me understand the issue in problem 2?

Thanks,
Andreas.

Answer 2

I am experiencing a similar frustration. When I run my calculations as a direct calculation or as a project, I get other results than if I run a Monte Carlo sim.

I tried attaching a couple of screenshots here, but am for some reason met with a 12,000 character length limit.