Number of events in Monte Carlo simulations for analysis of interference between radiocommunication systems

Abstract

Monte Carlo simulations are widely used in analysis of interference between radiocommunication systems. The method performs n random experiments, each called an event, snapshot, or trial, to estimate the statistics of some parameter of the problem (for instance, the interference level). There is no formalization about the number of events necessary to run a simulation. In practice, this is often done based on the intuition of the analyst about the model. In this paper, we propose how to choose a number of events for Monte Carlo simulations. We address this issue by understanding that the protection criteria for radiocommunication systems are usually related to a quantile of the result, and then calculating the number of events for which it is also possible to estimate a confidence interval for it. Results show that, for a confidence level of 99.9%, it is possible to estimate confidence intervals for p-quantiles for p up to 99.89% with 10000 events.

Index Terms
Confidence interval; Monte Carlo; number of events; spectrum sharing studies.

I. INTRODUCTION

Coexistence conditions between radiocommunication systems that operate in the same spectrum band or in adjacent band may be established to prevent co-channel or adjacent channel interference between them. This analysis can be done through theoretical or experimental studies, which complement each other. Although an experimental study is ideal, it is not always feasible - sometimes it is impracticable to test all interference scenarios to evaluate if the interference level caused by the incoming system is acceptable. Theoretical analysis, on the other hand, can assess a wide range of scenarios. Besides, theoretical studies commonly support experimental research.

Due to the complexity of the systems, deterministic solutions are useful to study worst-case scenarios. If a possibility of harmful interference comes out, a statistical analysis of the problem can be done. In the context of sharing and compatibility studies, the Monte Carlo method stands out. The Radiocommunication Sector of the International Telecommunication Union (ITU-R) even has some reports and recommendations on how to apply it [¹]-[³].

The Monte Carlo method runs several (n) independent random experiments on a model [⁴]. In each experiment, the algorithm randomly selects a scenario and simulates it. At the end of the process, it is possible to extract a probability density function (pdf) of the result and estimate statistics of the variables of interest.

In sharing and compatibility studies, the inputs are the propagation environment and features of the systems (e.g., antennas, receivers, emission masks), which are probabilistically defined through their pdf. For example, in [⁵] the interference between International Mobile Telecommunications (IMT) and TV receive-only (TVRO) systems in the C-Band in Brazil was studied and several statistical distributions were used: the propagation environment considers a gaussian distribution for the random component of the path loss; a binomial distribution models if the user is in an indoor or outdoor environment; a uniform distribution represents the distance between transmitter and receiver; constants describe the transmitter height and maximum transmitter power, etc. In each experiment, the Monte Carlo method samples these inputs and calculates the interfering signal level, which allows to estimate the percentage of scenarios where harmful interference occurs.

The result of the simulated model differs from the physical phenomenon. This difference can be caused by a poorly modeled system, or by an acknowledged error in any phase of the simulation, such as the statistical error of the estimation. This error can be reduced by increasing the sample size (n) and is the aim of this article.

Although the Monte Carlo method is widely used in interference analysis, there is no guidance on the number of samples required for the simulation. A typical value in sharing studies is n = 20000, which is the default value of SEAMCAT [⁶] (a software developed by the European Conference of Postal and Telecommunication administrations used for sharing studies between radiocommunications systems). But, depending on the complexity of the problem and the computational cost to simulate a single scenario, fewer values of n are used (e.g., 10000 or 5000). Occasionally, analysts rely on their intuition to chose n for their simulation, checking if the plotted result is smooth enough, even without a proper definition of "smooth enough".

We approach this issue considering that we are often interested in some quantile of the result. For example, during the ITU-R study period 2015-2019, an interference-to-noise ratio (I/N) less than 0 dB in at least 99.98% (percentage of time, probability or location) was proposed for the protection criteria at 24.65-25.25 GHz and 27.0-27.5 GHz for fixed-satellite services (FSS) and broadcasting-satellite services [⁷]. For the study period 2019-2023, an I/N less than -1.3 dB in at least 99.995% of the events was proposed for FSS systems at 3600 MHz-3800 MHz [⁸]. Using the desirable quantile of the I/N at the interfered-with systems (in these examples, 99.98% and 99.995%), it is possible to estimate an initial value of n to define a confidence interval for the quantile.

This article is structured in 5 sections. Next section reviews the Monte Carlo method and two methods to estimate confidence intervals for quantiles. Section 3 defines the minimum number of events for a Monte Carlo simulation. Section 4 discusses it in the context of two sharing studies. Finally, we present our conclusions.

II. REVIEW OF METHODS TO ESTIMATE CONFIDENCE INTERVAL FOR QUANTILES AND THE MONTE CARLO ALGORITHM

A. Monte Carlo method

According to the Report ITU-R SM. 2028-2, "the Monte Carlo method has been used for the simulation of random process and is based upon the principle of taking samples of random variables from their defined probability density functions" [¹]. In summary, a Monte Carlo simulation follows these steps:

1. Problem definition

   Initially, the interactions between the interfering and the interfered-with systems are modeled as Y = h(X₁,...,X_i,...,X_m), where X_i is one of the m random variables describing the features of the systems and the propagation environment, 1 ≤ i ≤ m, and Y is the output.

   The inputs X_i are defined by their probability density function (pdf: f_i(X)) or, equivalently, by their cumulative distribution function (cdf: F_i(X)).

   h(.) is a generic function that represents all interactions between the inputs to generate an output. It can be a mathematical function so simple that its computational implementation can be done in one line of code, or so complex that it needs to be implemented in a module calling other subfunctions.

2. Random experiments

   The method performs n random experiments (each one is called an event, snapshot, or trial). At each event j, for 1 ≤ j ≤ n, it samples the inputs according to their pdf and calculates y_j = h(x_1,j,...,x_i,j,...,x_m,j), where x_i,j and y_j are observed values of X_i and Y at the event j.

3. Analysis of the simulation results

   The simulation produces the sequence y = [y₁,...,y_j,...,y_n], which is used to estimate the pdf and cdf of Y (respectively, f_Y (Y ) and F_Y (Y )). By the law of large numbers, as n increases, the pdf of y approaches f_Y (Y ) [⁹], allowing to estimate parameters of Y (mean, standard deviation, quantiles, etc).

   Although a simulation can generate several outputs, this is not relevant to the topic discussed in this article. Therefore, just one output is considered.

Fig. 1 illustrates the steps above.

B. Confidence interval for quantiles

After the simulation of n interference scenarios, we are often interested in some quantile of the result. For example, if harmful interference is accepted in 0.1% of the possible scenarios, the interference signal level at the 99.9% quantile can be evaluated to check if there is harmful interference above the accepted level.

From now on, assume that y is an ordered sequence, i.e., the simulation result was sorted so that y_j ≤ y_k for any j < k. The p-quantile of a continuous distribution is defined as F_Y^-1(p) = sup{y : F_Y (y) ≤ p}, where sup is the supremum of a set [¹⁰]. This parameter can be estimated by the psample quantile (ξ_p), whose value is close to the np^th element of y. Since np is not always an integer, a consistent and asymptotically unbiased estimator for F_Y^-1(p) is y_(⌊np⌋+1) [¹⁰], [¹¹]. Other estimators use a linear interpolation of y_(⌊np⌋) and y_(⌊np⌋+1) [¹²]. Regardless of the estimator, y has ⌊np⌋ elements lower than or equal to ξ_p.

Fig. 1
At each event j, the Monte Carlo algorithm randomly samples all the m inputs according to their probability density function. Then, it calculates y_j = h(x_1,j,...,x_i,j,...,x_m,j). After n events, the pdf of Y can be estimated.

Due to the limited amount of sample data, there is a statistical uncertainty using ξ_p as the p-quantile. So, the estimation has an associated confidence interval. A confidence interval with confidence level of 100(1-α)% ¹ for any parameter θ means that "if one repeatedly calculates such intervals from many independent random samples, 100(1 - α)% of the intervals would, in the long run, correctly include the current value θ" [¹³]. The range itself is also estimated from limited sample data and sometimes it will not contain the parameter of interest. Typical values for the confidence level are 95% (α = 5%), 99% (α = 1%), and 99.9% (α = 0.1%).

The exact confidence interval for quantiles depends on the type of the statistical distribution, which is unknown in advance in this case. However, distribution-free nonparametric procedures can be used to calculate it [¹³]. The next sections review two of these methods.

1) Normal approximation to the binomial distribution: Consider the ordered sequence y = [y₁,...,y_n]. The p-sample quantile, ξ_p, indicates that y has ⌊np⌋ elements lower than or equal to ξ_p. A confidence interval for the p-quantile can be estimated using elements of y, namely y_r and y_s, where y_r < ξ_p < y_s.

To find r and s (positions in the sequence y), let’s map y to y^′, whose elements are defined as success or failure depending on if the corresponding element of y is lower than or equal to ξ_p or not. The process that generates y^′ can be described by a binomial distribution of parameters n and p, where p is the success rate (and it is also the quantile that we are interested in). The probability of exactly k successes in n trials is:

(1) $$Prob(k,n,p)=\frac{n!}{k!(n-k)!}{p}^k(1-p{)}^{n-k}$$

Note that the expected number of successes is np. A confidence interval that covers it with at least 100(1 - α)% probability can be specified finding r and s such that the probability of getting at least r and at most s success is greater than or equal to (1 - α):

(2) $$\sum _{k=r}^{s}Prob(k,n,p)=\sum _{k=r}^s\frac{n!}{k!(n-k)!}{p}^k(1-p{)}^{n-k}\ge 1-\alpha$$

Equation 2 shows that the binomial process with success rate p has a probability of at least 100(1- α)% of generating a number of successes between r and s in n events. Mapping this result back from y^′ to y, the confidence interval for the p-quantile with confidence level of 100(1-α)% is [y_r,y_s] [¹¹].

Although the confidence interval can be calculated using any choice of r and s that satisfies equation 2, it is reasonable to make the interval as small as possible [¹¹]. For np ≥ 5 and n(1 - p) ≥ 5, the binomial distribution can be approximated to a normal distribution with mean np and variance np(1-p). Then, r and s can be calculated as [¹⁴]:

(3) $$r=\lceil np-z\sqrt{np(1-p)}\rceil$$

(4) $$s=\lceil np-z\sqrt{np(1-p)}\rceil$$

where z is the quantile function of the standard normal distribution calculated at (1 - α/2). The normal approximation to the binomial distribution will generate non-integer indexes. Therefore, it is necessary to round them up or down to an integer. A conservative confidence interval is obtained rounding r down and rounding s up. Nevertheless, a compromise is to round both r and s in the same direction [¹⁴], as in equations 3 and 4. Moreover, since the normal distribution has a maximum at np and is symmetric around it, equations 3 and 4 assume that the distances from the indexes r and s to ⌊np⌋ are approximately symmetric. However, this doesn’t imply that the confidence interval of y will also be symmetric around ξ_p. Fig. 2 illustrates the calculation of r and s using equations 3, and 4.

Fig. 2
Example on how to estimate r and s for a simulation with n = 1000, p = 95%, and α = 5%. The circles show the probability mass function for the binomial distribution. The line shows the probability density function for its normal approximation. The confidence interval for the binomial distribution (red circles) is calculated as [r,s] = [936,963]. Its normal approximation estimates it (red line) as [r,s] = [937,964]. The x-axis was truncated at [923,973].

2) Bootstrap: Another way to calculate the confidence interval is using an algorithm from the bootstrap family. There are some alternatives, and all of them are based on the basic principle of resampling the output of the simulation. This section describes the bootstrap percentile algorithm [¹⁵], which is presented in Fig. 3.

The output of the simulation is the sample y = [y₁,...,y_n] of size n, which is resampled with replacement to generate B bootstrap samples, ${\mathbf{y}}^{\mathbf{\ast }}=[{\mathbf{y}}_{\mathbf{1}}^{\mathbf{\ast }},...,{\mathbf{y}}_{\mathbf{k}}^{\mathbf{\ast }},...,{\mathbf{y}}_{\mathbf{B}}^{\mathbf{\ast }}]$, each one with size n: ${\mathbf{y}}_{\mathbf{k}}^{\mathbf{\ast }}=[y_{k,1}^{\ast },...,y_{k,n}^{\ast }]$. Some function t(.) is applied to each bootstrap sample to extract an estimation of the statistic of interest θ of the boostrap sample (in this case, the p-quantile). The result is a set of B bootstrap replications of the statistic of interest: ${\mathit{\theta }}^{\mathbf{\ast }}=[{\theta }_1^{\ast },...,{\theta }_B^{\ast }]$, where ${\theta }_k^{\ast }=t({\mathbf{y}}_{\mathbf{k}}^{\mathbf{\ast }})$.

The bootstrap estimate of θ is the mean of the B replications of the statistic of interest, i.e, the mean of $[{\theta }_1^{\ast },...,{\theta }_B^{\ast }]$. Moreover, the algorithm calculates its confidence interval with 100(1-α)% confidence level as the α/2 and (1 - α/2) quantiles of θ^∗. B must be at least 2000 [¹⁵].

Fig. 3
Bootstrap replication of the statistic of interest.

III. MINIMUM NUMBER OF EVENTS (n) IN A MONTE CARLO SIMULATION TO ESTIMATE A CONFIDENCE INTERVAL FOR A QUANTILE OF THE RESULT.

The previous section presented two methods to calculate confidence intervals for quantiles. The first relies on the normal approximation to the binomial distribution. The latter is based on the concept of resampling the data and aims for wide applicability with near-exact accuracy [¹⁵], which encourages its use in the estimation of confidence intervals for the parameter of interest.

Both methods estimate the confidence interval within the data. However, the normal approximation method helps to define a minimum value of n. s and r are positions of elements of the sample and are calculated considering that the distance from r and s to np are approximately equal. Depending on the value of n, equation 4 may produce s > n, which implies that it is not possible to get y_s from the data. So, a constraint is that s ≤ n, i.e., s = n - a for any a ∈ [0,n/2). The minimum number of events (n_min) is derived from equation 4 (see the Appendix) and depends on p, α, and a:

(5) $$n_{min}(p,\alpha ,a)=\lceil \left(\frac{1}{1-p}\right){\left(\frac{z\sqrt{p}+\sqrt{z^{2}p+4a}}{2}\right)}^2\rceil$$

Equation 5 assumes p > 0.5; however, due to the symmetry of the normal distribution, the results for the p-quantile also apply to the (1-p)-quantile. Additionally, n_min must also satisfy the previously defined constraints (np ≥ 5 and n(1 - p) ≥ 5).

For $a=0,n_{min}=\lceil z\sqrt{p}/(1-p)\rceil ,s=n_{min}$, and y_s = max(y), i.e., the upper bound of the confidence interval is also the largest value of y.

If a > 0, the upper boundary of the confidence interval will be y_s = y_n-a, meaning y_s will be the (a+1)^th largest element of y. This value acts as a safety margin: by setting a > 0, the analyst increases the value of n_min to exclude the a^th largest elements of y from the confidence interval calculated by equation 4. This approach can be useful to remove up to a outliers from the confidence interval.

TABLE I
Minimum number of events to estimate the upper bound of the confidence interval as y_s = y_n-a.

	Confidence level (%) for a = 0			Confidence level (%) for a = 1
p quantile (%)	95	99	99.9	95	99	99.9
95	100	127	206	110	164	245
99	500	657	1072	563	846	1265
99.9	5000	6629	10817	5661	8511	12739
99.98	25000	33168	54128	28321	42581	63735
99.99	50000	66343	108265	56646	85169	127481
99.995	100000	132692	216541	113295	170344	254972
99.999	500000	663484	1082746	566490	851743	1274903

TABLE II
Maximum p-quantile in which it is possible to estimate the confidence interval using equation 4 for a = 0 and α = 0.1%.

Number of events (n)	Maximum p quantile (%)
1000	98.92
3000	99.46
5000	99.78
10000	99.89
20000	99.94
50000	99.97

Table I shows the value of n_min for a = 0 or 1 and some useful combinations of α and p. Table II shows the maximum p-quantile that can be estimated for a given sample size n. However, they do not indicate the length of the confidence interval. Therefore, our advice is to simulate with at least n_min events, then calculate the confidence interval and check if it is adequate for the scenario under analysis. If the estimated value plus error exceeds the protection criteria, it is appropriate to increase the sample size to narrow the confidence interval and to reduce uncertainty. Knowing that the length of the confidence interval for quantiles usually decreases with 1/ $\sqrt{n}$ [¹³], [¹⁵], increasing n by a factor of k generally reduces the width of the confidence interval by approximately $\sqrt{k}$ (it is not exactly $\sqrt{k}$ because the length of the confidence interval is itself random) [¹³]. As Table II shows, n = 20000 (the default number of samples of SEAMCAT [⁶]) is sufficient to estimate a confidence interval for quantiles up to 99.94% with confidence level up to 99.9%.

IV. DISCUSSION

This section applies the proposal presented in this paper to three previously published sharing studies. The first example considers the potential overloading caused by an IMT base station (BS) at TVRO receivers in Brazil [⁵] at 3.5 GHz. These results not only supported experimental tests but also facilitated the discussions during the 3.5 GHz spectrum auction.

The other examples present two sharing studies between High-Altitude Platform Stations as IMT Base Stations (HIBS) and other radiocommunication systems. High-Altitude Platform Stations are defined as "radio stations located on an object at an altitude of 20-50 km and at a specified, nominal, fixed point relative to the Earth" [¹⁶]. Equipped with IMT base stations, these platforms can provide mobile services over a large area and are useful for complementing IMT terrestrial coverage [¹⁷]. Due to its extensive coverage, there is a risk that this new system causes harmful interference in existing systems. Therefore, sharing studies involving HIBS have been done in the last few years [¹⁸]. Both studies were submitted to the ITU-R as part of the Brazilian contribution [¹⁹], [²⁰] and were approved to be part of the chairman’s report, as supporting documents to the conference preparatory meeting report [²¹].

The focus of this paper is on analyzing the number of events in a Monte Carlo simulation. The examples have been selected from previous studies to illustrate how the number of events should be considered in Monte Carlo simulations. Hence, only a brief description of the coexistence problem will be provided, and the reader might refer to the original studies for detailed descriptions of the problems [⁵], [¹⁸].

A. Sharing study between IMT base station operating at 3560-3600 MHz and TVRO receivers operating at 3625-4200 MHz

This example illustrates a situation that occurred in Brazil before the introduction of 5G in the 3.5 GHz band. The study assessed the impact of the interference from IMT base stations operating in the 3560-3600 MHz frequency band on TVRO receivers installed in the 3625-4200 MHz frequency band (C-band). In Brazil, most of the low noise block downconverters (LNB) of TVRO receivers lacked a C-band filter. Therefore, Monte Carlo simulations were used to determine whether the IMT base station emissions would overload the TVRO receivers operating in an adjacent band.

In this simulation, the IMT base station height is 20 m and the transmitter antenna is tilted at -10^◦. The effective isotropic radiated power is 46 dBm, subject to the 3GPP emission mask [²²]. Each cell is divided into three sectors and has a reuse factor of 1. The radius of each cell is 150 m (17.1 BS/km²). The TVRO receivers are distributed around the cell keeping a minimum separation distance of 20 m from the nearest base station. The height of the receiver antenna is 6 m, and its radiation pattern is modeled using [²³]. The propagation environment between the IMT base station transmitter and the TVRO receivers is based on [²⁴]. The overload threshold (O_th) of the receivers is -45 dBm.

The original study [⁵] considered that an acceptable percentage of TVRO receivers affected by the IMT system is 5%, i.e., it is necessary to verify the intensity of the IMT signal received at all TVRO receivers and analyze if the interference level at the 95% quantile is lower than the overload threshold. The original simulation used 20000 events. At the end of the simulation, the interference signal level the 95% quantile was -50.97 dBm, and since it was lower than -45 dBm, this scenario was considered adequate.

Table I shows that at least 206 events are necessary to estimate a confidence interval with 99.9% confidence level. Thus, the simulation could have initially been conducted with only 206 events, followed by further analysis to determine if more events were necessary. To illustrate this, Fig. 4 shows the cdf of I/N at the TVRO receivers calculated after 206 and 20000 events. In both cases, the 95% quantile (-51.12 dBm for n = 206 and -50.97 dBm for n = 20000) satisfies the criterion of O_th < -45 dBm. Usually, it is assumed that the higher the value of n, the better. However, before drawing this conclusion, it is necessary to check the confidence interval of the estimated parameters. Even with a significant wide interval, a low value of n might be sufficient to draw the desired conclusion from the analysis.

Fig. 5 shows the estimated 95% quantile and its confidence interval with 99.9% confidence level for the first n events, where n = 206,1000,2000,...,20000. Although the confidence interval for n = 206 is significantly wider than the one for n = 20000, the upper bound of the confidence interval does not exceed the O_th = -45 dBm. For the simulated scenario, 206 events would have been sufficient to conclude for a non-interference scenario, considerably fewer than the number of simulations used in the original study.

Fig. 4
Cdf of the out-of-band interference level at the TVRO receivers calculated after 206 and 20000 events.

Fig. 5
Estimated 95% quantile of the out-of-band interference at the TVRO receivers and its 99.9% confidence interval calculated using the normal approximation and the boostrap percentile method. The x-axis starts at n = 206.

B. Sharing study between HIBS operating at 1710-1850 MHz and aeronautical mobile service (AMS) operating at 1780-1850 MHz

AMS systems are used for airborne data links, including video to support remote sensing, earth sciences, etc. The link is established between a terminal carried by an aircraft and a ground terminal [²⁵]. Fig. 6 shows the interfering scenario between the HIBS platform and a ground station. The platform covers an area with 200 km diameter. The aircraft is uniformly distributed over a 800 km radius below the HIBS nadir, and its altitude is uniformly distributed from 38000 ft (11582 m) to 58070 ft (17700 m). The aircraft carries the AMS airborne antenna positioned at the bottom of the airplane and establishes a link with a ground terminal uniformly distributed over a 500 km radius below the HIBS nadir. The propagation loss considers diffraction loss, clutter loss, and attenuation due to the fuselage of the aircraft [²⁶]-[²⁹]. Although several scenarios were simulated, this section shows the one where the ground terminal uses a directional antenna (whose azimuth was uniformly distributed over 0^◦ to 360^◦) and the HIBS is operating at an altitude of 20 km. The I/N was calculated at the ground terminal, whose protection criteria is -6 dB. A detailed description of the problem is available in [¹⁸], [¹⁹].

Fig. 6
HIBS and AMS: Interference scenario.

Consider that no more than 1% of the scenarios can suffer harmful interference, i.e., the 99% quantile of the I/N at the ground terminal should be lower than -6 dB. According to Table I, at least 657 events are necessary to estimate a confidence interval with 99% confidence level. The original simulation used 30000 events, whose cdf is shown in Fig. 7. The I/N at the 99% quantile is -9.54 dB, lower than the protection criteria.

Fig. 7
Cdf of I/N at the ground terminal after 30000 events.

However, as it is a sample quantile, this estimate should be associated with its confidence interval. Fig. 8 shows the sample quantile with its 99% confidence interval for the first n events, where n = 1000,2000,...,30000.

The first thing to note is that, for n = 1000, there is a considerable divergence in the estimations of the upper limit of the confidence interval calculated using the bootstrap and the normal approximation. Nothing special is happening. The simulation (n = 1000) generated 3 points with a relatively high I/N, which pushed the upper limit of the confidence interval calculated by the normal approximation upwards. In the bootstrap method this effect was softened due to the resampling.

Fig. 8
Estimated 99% quantile of the I/N at the ground terminal and its 99% confidence interval calculated using the normal approximation and the bootstrap percentile method. The x-axis starts at n = 1000.

Suppose only 1000 events were used to simulate and that the normal approximation method was used to estimate the 99% confidence interval. In this case, the I/N at the 99% quantile is -7.49 dB, lower than the protection criteria, but the confidence interval is [-11.45,-0.79] dB (length: 10.66 dB), which includes the protection criteria. Therefore, the uncertainty associated with the sample quantile does not allow us to conclude that it complies with the protection criteria. Thus, it is recommended to narrow the confidence interval. Since its length is proportional to 1/ $\sqrt{n}$, if we double the number of events, it is expected that the length of the confidence interval would be reduced by a factor of approximately 1/$\sqrt{2}$ (70%). In this example, the confidence interval for n = 2000 is [-13.23,-6.27] dB (length: 6.96 dB), which is 65% of the length for n = 1000.

This can be seen also for the bootstrap. For example, the confidence interval calculated for n = 5000 and n = 30000 are [-11.5,-7.88] dB (length: 3.62 dB) and [-10.15,-8.52] dB (length: 1.63 dB). This is a reduction factor of 45%, close to the expected one (1/ $\sqrt{6}$ = 41%).

Despite the original simulation using 30000 events, the same conclusion could be obtained with far fewer events. Regardless of the method used to estimate the confidence interval, only 2000 events would provide the necessary accuracy to conclude that, with a confidence level of 99%, no harmful interference occurs in more than 1% of the scenarios.

The number of simulations that allowed for a conclusion of a non-interference scenario (n = 2000) is higher than initially calculated (n = 657). As discussed in Section III, the value of n_min indicates the minimum number of simulations to estimate a confidence interval for the p-quantile. It does not relate to the maximum error in the analysis. In some instances, as in the first example, this minimum number of simulations will suffice to determine the presence or absence of harmful interference. However, in this example, the confidence interval includes both scenarios. When this occurs, as each new event in the Monte Carlo method is independent of the previous ones, it is necessary to add more events until the estimate and its confidence interval support a valid conclusion.

C. Sharing study between HIBS operating at 694-960 MHz and broadcasting services at 767 MHz(Channel 63)

In this scenario, the HIBS and the broadcasting network are located at the borders of two different countries. The HIBS is at an altitude of 20 km covering an area of 200 km diameter of one country and may interfere in the digital terrestrial television broadcasting receivers (DTTB) that are uniformly distributed within an area of 40 km radius in another country. The protection criteria for the DTTB receivers is I/N less than -10 dB, and is acceptable that 5% of the scenarios might suffer harmful interference [¹⁹]. The propagation loss between the HIBS and the DTTB receivers considers absorption by gases, diffraction and clutter losses [²⁶]-[²⁸]. Some mitigation measures were proposed for harmonious coexistence between both services (the transmitted power of the HIBS was reduced in 3 dB and a separation distance of 80 km between the two coverage borders was set), which were validated with a Monte Carlo simulation with 40000 events. Fig. 9 illustrates the interference scenario.

Fig. 9
HIBS and DTTB: Interference scenario.

Table I shows that at least 127 events are necessary to estimate a confidence interval for the 95% quantile with a 99% confidence level. Fig. 10 displays the confidence interval for the first n events, where n = 127,1000,2000,...,40000. For 127 events, the I/N at the 95% quantile (-7.64 dB) does not comply with the protection criteria. However, since the confidence interval associated with this estimate includes the protection criterion, there is still not enough data to conclude whether the level of interference complies with the protection criteria - more data are necessary to reduce the uncertainty. For n = 1000 events, the I/N at the 95% quantile (-10.78 dB) complies with the protection criteria, but the confidence interval still includes it. Only for n = 2000 the confidence interval narrows enough to fall below the protection criterion, i.e., 2000 events are sufficient to conclude that, with a confidence level of 99%, there is harmful interferences in less than 5% of the scenarios.

Fig. 10
Estimated 95% quantile of the I/N at DTTB receiver and its 99% confidence interval calculated using the normal approximation and the bootstrap percentile method. The x-axis starts at n = 127.

In many sharing studies, there is no clear reason when choosing the number of events and it is common to simulate until the plotted result is "smooth enough". Fig. 11 shows the cdf between 85% and 100% for 2000 and 40000 events. For this zoom level, the plot for n = 40000 is smoother than the one for n = 2000. But if analysts are guided by how smooth is the plotted line, they can be tricked by the zoom level of the plotted result. Besides, it might cause an unnecessary excess of simulations, which can be time-consuming depending on the complexity of the simulation. What should guide the number of events is the level of certainty necessary for the simulated scenario.

Fig. 11
Cdf of I/N at the DTTB receiver after 2000 and 40000 events.

V. CONCLUSIONS

Telecommunications administrations, industry, scientific sectors, and regulatory bodies regularly publish sharing and compatibility studies using the Monte Carlo method. However, many studies rely on the analyst’s intuition to define the number of events (n) for the simulations. Often, this is done by checking the plot of the probability density function of the output and adding new events until it visually converges.

This strategy has some disadvantages. Relying on intuition or visual convergence of the output often results in more simulations than necessary. This becomes particularly concerning as coexistence problems grow more complex and computationally intensive. Moreover, there is no concern for presenting the results with a margin of error.

In this article, we present a proposal to choose the number of events for a Monte Carlo simulation. A typical value used in sharing and compatibility studies is n = 20000, which allows the analysis of the p-sample quantile and its 99.9% confidence interval for p up to 99.94%. For n = 1000 and n = 10000 with the same confidence level, p can be up to, respectively, 98.92% and 99.89%.

This number of events is a starting point to simulate and calculate a confidence interval for the desired quantile. Depending on the characteristics of the problem (and this will only become clear after the first simulation, especially if the estimated value is very close to the protection criteria or if the sampling error is relatively large), new simulations must be added to the initial one.

In the long run, a Monte Carlo simulation can generate very specific combinations of input parameters values, resulting in outcomes that are extreme outlier. The methodology discussed in this article does not support conclusions regarding these rare data that were not observed in the sample. The Python code to generate all the results of this article (tables, figures, and equations) is available at [³⁰].

• 1
  The confidence interval with a confidence level of 100(1 - α)% is also called a 100(1 - α)% confidence interval.

APPENDIX

A. Derivation of equation 5

Equation 5 is derived from equation 4 setting s = n - a and solving for n:

(6) $$n-a=\lceil np+z\sqrt{np(1-p)}\rceil$$

Since $n-a\in \mathbb{N}$:

(7) $$\lceil np+z\sqrt{np(1-p)}-n+a\rceil =0$$

Setting $w=\sqrt{n(1-p)}$ and w² = n(1 - p):

(8) $$\lceil -w^2+z\sqrt{p}w+a\rceil =0$$

Due to the ceil function in equation 8, the argument of the function should be in (-1,0] interval, i.e., it is valid near the roots of the argument of the ceil function. The roots are:

(9) $$w_{roots}=\frac{z\sqrt{p}\pm \sqrt{z^{2}p+4a}}{2}$$

Since a ∈ [0,n/2), min(w_roots) ≤ 0 and, consequently, n ≤ 0, an invalid result. So, max(w_roots) is the only valid root of equation 9.

Considering that the argument of the ceil function in equation 8 is a downward-facing parabola and that it must be in the interval (-1,0], thus w ≥ max(w_roots) subject to equation 8. As we are interested in the lowest value of n (n_min) and w ∝ $\sqrt{n}$, then w = max(w_roots):

(10) $$max(w_{roots})=\sqrt{n_{min}(1-p)}=\frac{z\sqrt{p}+\sqrt{z^{2}p+4a}}{2}$$

Solving for n_min and considering that it should be rounded up to the next integer:

(11) $$n_{min}=\lceil \left(\frac{1}{1-p}\right){\left(\frac{z\sqrt{p}+\sqrt{z^{2}p+4a}}{2}\right)}^2\rceil$$

REFERENCES

• ^[1]
  ITU, “Report ITU-R SM.2028-2: Monte Carlo simulation methodology for the use in sharing and compatibility studies between different radio services or systems,” International Telecommunication Union, Geneva, Switzerland, Tech. Rep., 2017. [Online]. Available: https://www.itu.int/pub/R-REP-SM.2028-2-2017
  » https://www.itu.int/pub/R-REP-SM.2028-2-2017
• ^[2]
  --, “Recommendation ITU-R M.1634: Interference protection of terrestrial mobile service systems using Monte Carlo simulation with application to frequency sharing,” International Telecommunication Union, Geneva, Switzerland, Tech. Rep., 2003. [Online]. Available: https://www.itu.int/rec/R-REC-M.1634-0-200306-I
  » https://www.itu.int/rec/R-REC-M.1634-0-200306-I
• ^[3]
  --, “Recommendation ITU-R M.2101: Modelling and simulation of IMT networks and systems for use in sharing and compatibility studies,” International Telecommunication Union, Geneva, Switzerland, Tech. Rep., 2017. [Online]. Available: https://www.itu.int/rec/R-REC-M.2101-0-201702-I
  » https://www.itu.int/rec/R-REC-M.2101-0-201702-I
• ^[4]
  D. P. Kroese, T. Taimre, and Z. I. Botev, Handbook of Monte Carlo methods, 1st ed. New Jersey: Wiley, 2011.
• ^[5]
  L. C. Fernandes, A. Linhares, “Coexistence conditions of LTE-advanced at 3400-3600 MHz with TVRO at 3625-4200 MHz in Brazil,” Wireless Networks, vol. 25, pp. 105-115, 2019. [Online]. Available: https://doi.org/10.1007/s11276-017-1544-8
  » https://doi.org/10.1007/s11276-017-1544-8
• ^[6]
  CEPT, “ECC report 252: SEAMCAT handbook,” International Telecommunication Union, Montreux, Switzerland, Tech. Rep., 2016. [Online]. Available: https://docdb.cept.org/download/1270
  » https://docdb.cept.org/download/1270
• ^[7]
  ITU, “Document 5-1/411-E. Reply liaison statement to ITU-R working party 5D - WRC-19 agenda item 1.13 (IMT),” International Telecommunication Union, Tech. Rep., 2018. [Online]. Available: https://www.itu.int/md/R15-TG5.1-C-0411
  » https://www.itu.int/md/R15-TG5.1-C-0411
• ^[8]
  --, “Document 5D/734-E. Reply liaison statement to ITU-R Working Party 5A - Technical and operational characteristics and protection criteria of FSS systems for sharing and compatibility studies on WRC23 agenda item 1.3,” International Telecommunication Union, Tech. Rep., 2021. [Online]. Available: https://www.itu.int/md/R19-WP5A-C-0395
  » https://www.itu.int/md/R19-WP5A-C-0395
• ^[9]
  S. Hartman, Ed., Handbook of the history of logic - Volume 10: inductive logic, 1st ed. Amsterdam: Elsevier, 2011.
• ^[10]
  B. C. Arnold, N. Balakrishnan, and H. N. Nagaraja, A first course in order statistics, 1st ed. New York: Society for Industrial and Applied Mathematics, 2008.
• ^[11]
  H. A. David and H. N. Nagaraja, Order statistics, 3rd ed. New Jersey: Wiley, 2003.
• ^[12]
  R. J. Hyndman and Y. Fan, “Sample quantile in statistical packages,” American Statiscal Association, vol. 50, no. 4, pp. 361-365, 1996. [Online]. Available: https://doi.org/10.2307/2684934
  » https://doi.org/10.2307/2684934
• ^[13]
  W. Q. Meeker, G. J. Hahn, and L. A. Escobar, Statistical intervals - a guide for practioners and researchers, 2nd ed. New Jersey: Wiley, 2017.
• ^[14]
  W. J. Conover, Pratical Nonparametric Statistics, 3rd ed. New York: Wiley, 1999.
• ^[15]
  B. Efron and T. Hastie, Computer age statistical inference: algorithms, evidence, and data science, 1st ed. New York: Cambridge University Press, 2016.
• ^[16]
  ITU, “Radio Regulations - Articles,” International Telecommunication Union, Geneva, Switzerland, Tech. Rep., 2020. [Online]. Available: http://handle.itu.int/11.1002/pub/814b0c44-en
  » http://handle.itu.int/11.1002/pub/814b0c44-en
• ^[17]
  L. C. Alexandre, A. Linhares, G. Neto, and A. C. Sodre, “High-Altitude Platform Stations as IMT Base Stations: Connectivity from the Stratosphere,” IEEE Communications Magazine, vol. 59, no. 12, pp. 30-35, 2021. [Online]. Available: https://doi.org/10.1109/MCOM.001.2100477
  » https://doi.org/10.1109/MCOM.001.2100477
• ^[18]
  L. C. Alexandre, “HIBS and IMT-2020 Coexistence with other Space/Terrestrial Communications and Radar Systems,” Ph.D. dissertation, Inatel, Santa Rita do Sapucaí, MG, Brazil, 2022. [Online]. Available: https://biblioteca.inatel.br/cict/acervopublico/sumarios/TesesdeDoutoradodoInatel/LucianoCamiloAlexandre.pdf
  » https://biblioteca.inatel.br/cict/acervopublico/sumarios/TesesdeDoutoradodoInatel/LucianoCamiloAlexandre.pdf
• ^[19]
  Brazil, “Document ITU-R WP5D. Contribution 858. Updates on sharing study between broadcasting services and HIBS in band 1 in Annex 1 of the working document towards sharing and compatibility studies of HIBS under WRC-23 agenda item 1.4,” International Telecommunication Union, Tech. Rep., 2022. [Online]. Available: https://www.itu.int/md/R19-WP5D.AR-C-0858
  » https://www.itu.int/md/R19-WP5D.AR-C-0858
• ^[20]
  --, “ITU-R WP5D Contribution 857. New proposed sharing study between AMS and HIBS in band 2 to be added to the Annex 2 of the working document towards sharing and compatibility studies of HIBS under WRC-23 agenda item 1.4,” International Telecommunication Union, Tech. Rep., 2022. [Online]. Available: https://www.itu.int/md/R19-WP5D.AR-C-0857
  » https://www.itu.int/md/R19-WP5D.AR-C-0857
• ^[21]
  ITU, “Report of the CPM on technical, operational and regulatory/procedural matters to be considered by the World Radiocommunication Conference 2023,” International Telecommunication Union, Tech. Rep., 2023. [Online]. Available: https://www.itu.int/md/R19-CPM23.2-R-0001/en
  » https://www.itu.int/md/R19-CPM23.2-R-0001/en
• ^[22]
  3GPP, “LTE; Evolved Universal Terrestrial Radio Access (E-UTRA); Base Station (BS) radio transmission and reception (3GPP TS 36.104 version 13.2.0 Release 13) ,” 3rd Generation Partnership Project, , Tech. Rep., 2016. [Online]. Available: https://www.etsi.org/deliver/etsi_ts/136100_136199/136104/13.02.00_60/ts_136104v130200p.pdf
  » https://www.etsi.org/deliver/etsi_ts/136100_136199/136104/13.02.00_60/ts_136104v130200p.pdf
• ^[23]
  ITU, “Recommendation ITU-R S.465-6: Reference radiation pattern for earth station antennas in the fixedsatellite service for use in coordination and interference assessment in the frequency range from 2 to 31 GHz,” International Telecommunication Union, Geneva, Switzerland, Tech. Rep., 2010. [Online]. Available: https://www.itu.int/rec/R-REC-S.465-6-201001-I
  » https://www.itu.int/rec/R-REC-S.465-6-201001-I
• ^[24]
  --, “Recommendation ITU-R P.452-18: Prediction procedure for the evaluation of interference between stations on the surface of the Earth at frequencies above about 100 MHz,” International Telecommunication Union, Geneva, Switzerland, Tech. Rep., 2023. [Online]. Available: https://www.itu.int/rec/R-REC-P.452-18-202310-I
  » https://www.itu.int/rec/R-REC-P.452-18-202310-I
• ^[25]
  --, “Recommendation ITU-R M.2116: Technical characteristics and protection criteria for the aeronautical mobile service systems operating within the 4 400-4 990 MHz frequency range,” International Telecommunication Union, Geneva, Switzerland, Tech. Rep., 2018. [Online]. Available: https://www.itu.int/rec/R-REC-M.2116-0-201801-I
  » https://www.itu.int/rec/R-REC-M.2116-0-201801-I
• ^[26]
  --, “Recommendation ITU-R P.1409-2: Propagation data and prediction methods for systems using high altitude platform stations and other elevated stations in the stratosphere at frequencies greater than about 0.7 GHz,” International Telecommunication Union, Geneva, Switzerland, Tech. Rep., 2023. [Online]. Available: https://www.itu.int/rec/R-REC-P.1409-3-202308-I
  » https://www.itu.int/rec/R-REC-P.1409-3-202308-I
• ^[27]
  --, “Recommendation ITU-R P.619-5: Propagation data required for the evaluation of interference between stations in space and those on the surface of the Earth,” International Telecommunication Union, Geneva, Switzerland, Tech. Rep., 2021. [Online]. Available: https://www.itu.int/rec/R-REC-P.619-5-202109-I
  » https://www.itu.int/rec/R-REC-P.619-5-202109-I
• ^[28]
  --, “Recommendation ITU-R P.2108-1: Prediction of clutter loss ,” International Telecommunication Union, Geneva, Switzerland, Tech. Rep., 2021. [Online]. Available: https://www.itu.int/rec/R-REC-P.2108-1-202109-I
  » https://www.itu.int/rec/R-REC-P.2108-1-202109-I
• ^[29]
  --, “Recommendation ITU-R P.528-5: A propagation prediction method for aeronautical mobile and radionavigation services using the VHF, UHF and SHF bands,” International Telecommunication Union, Geneva, Switzerland, Tech. Rep., 2021. [Online]. Available: https://www.itu.int/rec/R-REC-P.528-5-202109-I
  » https://www.itu.int/rec/R-REC-P.528-5-202109-I
• ^[30]
  L. C. Fernandes, A. Linhares, and L. C. Alexandre. (2023) Code for ’Number of events in Monte Carlo simulations for analysis of interference between radiocommunication systems’. [Online]. Available: https://github.com/carisio/monte-carlo-n-min
  » https://github.com/carisio/monte-carlo-n-min

Publication Dates

• Publication in this collection
  19 Aug 2024
• Date of issue
  2024

History

• Received
  02 Nov 2023
• Reviewed
  04 Nov 2023
• Accepted
  19 June 2024