Rand Simulation Solutions Blog

Investigating Spontaneous Feline Antenna Loading with Hybrid Regions

As winter snowfall blanketed the United States over the past holiday season, many people gathered on the couch to stay cozy in their houses and stream some holiday favorites. However, for one cat owner who also happened to be a satellite internet customer, their day of watching movies at home may have been cut short:

 

Fig 1: A group of cats enjoying the self-heating feature of the Starlink dish designed to melt snow. Aaron Taylor via Twitter.

 

Twitter user Aaron Taylor posted the picture above on New Year’s Eve 2021, which shows 5 cats nestled together on top of the Starlink antenna, which happens to have a heating feature originally designed to melt snow which would otherwise build up on top of the dish. Starlink is a satellite internet service operated by SpaceX, with the professed goal of providing high-bandwidth internet service anywhere on the globe using a constellation of many hundreds of small satellites in low-earth orbit. While this picture seems to show that this feature works to prevent the accumulation of snow, to everyone’s surprise, it seems to have encouraged an accumulation of cats, which seemed to degrade the signal!

 

Fig 2: User reporting having internet bandwidth impacted by the presence of feline loading

 

Having a background in RF/Microwave engineering and the use of ANSYS HFSS, I had to know how the presence of this furry pileup would impact the performance of the antenna and thought it would be the perfect opportunity to demonstrate the use of HFSS’ Hybrid solver.

To simulate this situation, we’ll first need a dish antenna that operates over the entire range specified for the Starlink dish -- 10.7 GHz to 30 GHz. To stand-in for the actual Starlink antenna – which is a phased array of a couple of hundred small aperture-coupled patch antennas worthy of a blog post itself –, I’ll be using a more traditional horn-fed parabolic dish reflector antenna, generated with the Antenna Toolkit ACT extension included in HFSS. The Antenna Toolkit is a very useful extension that allows the user to synthesize any basic antenna for an arbitrary design frequency in seconds, which you can check out in the patch antenna synthesis example video seen here. The default settings were used, apart from setting the center frequency to 17.3 GHz, and the “A” and “B” dimensions to 29 cm to match the frequency range and diameter of the Starlink dish.

Fig 3: Parabolic reflector antenna matching approximate dimensions of Starlink dish, with center frequency 17.3 GHz

              

The antenna toolkit has set up this dish with Hybrid Regions, a very powerful capability within HFSS that allows users to use different solver types in parallel to solve one geometry, resulting in solution times orders of magnitude faster and more memory efficient than using the FEM solver alone. FEM stands for Finite Element Method and is the default solver within HFSS. This solver subdivides the solution volume containing given geometry into many tetrahedral mesh elements, at each of which discretized forms of Maxwell’s equations are enforced.  The remaining non-FEM solvers are integral equation (IE solver) or raytracing based (PO, SBR+ solvers), and can solve electrically large open scattering, and reflector problems very quickly compared to FEM. An arbitrary number of FEM and non-FEM regions can be combined in one hybrid solution.

Below, the two types of solutions which comprise the hybrid solve, and the geometry being used with each solver is shown – the horn feed antenna inside of a small FEM volume, and the reflector unbounded and designated as a non-FEM hybrid region:

 

 

4: FEM and Non-FEM solution regions in Antenna Toolkit generated hybrid solutions

 

By default, Antenna Toolkit has set up the reflector design to use the IE solver on the reflector. Examining the properties of this design, another solver might be more appropriate which brings us to the question of when to use each of the unbounded, non-FEM solvers available to us: IE, PO, or SBR+.

Over the course of our experiences using hybrid simulations at Rand, we have developed a few guidelines for when to use a particular solver. The first question is when to use a hybrid solution. Hybrid solutions are appropriate when there are electrically large volumes of free space in which you aren’t interested in seeing the specific E and H field values. Examples include multi-antenna studies, antenna placement studies, scattering, and radar cross-section simulations.  

When it is appropriate to use the hybrid solver, ask the following questions for each object in the hybrid solution: If an object to be simulated is less than 10 wavelengths long at the highest frequency in the solution (λ_min ) the FEM solver can be used. If the largest dimension of the non-FEM geometry is less than 30, use the IE solver. If greater than 30, use one of the raytracing solvers (PO or SBR+). If the object will only require one ray bounce to accurately simulate (single reflector designs, RCS, and antenna placement on convex objects) use PO, if it will require many bounces (all other applications) use SBR+. SBR+ has the added benefit of allowing the user to plot the rays fired as part of the simulation, as well as more fine control over the number of rays fired, so err on the side of using SBR+ if there is ambiguity between whether to use it or the PO solver. Details for the other two hybrid region types (FE-BI and Dielectric Cavity) can be seen tabulated in Appendix 1.

Fig 5: Flowchart detailing decision tree for deciding what solver to use in a hybrid solution

Fig 6: PO v.s. IE calculated radar cross-sections of a sphere with varying diameters. It can be seen how the responses converge past 30 lambdas, under which PO should not be trusted to have the most physical response.

 

In our case, we have a single reflector antenna, which is around 58 cm in diameter.  Our highest frequency in our simulation range is 30 GHz, so our minimum free-space wavelength is (c/3e10) = 0.01 m = 1cm. This puts us in the > 30*λ_min range, and since only one reflection is needed, the PO solver is most appropriate for our case. To reassign, I deleted the hybrid region corresponding to the dish and assigned a PO hybrid region to the sheet representing the dish.

Fig 7: Deleting IE region assigned by the Antenna Toolkit and re-assigning a PO region

 

 

With this, the cat-free dish antenna is ready to simulate. With default solver settings, the solution took 4 minutes and 50 seconds to simulate, with 178k matrix elements. As a point of comparison, I set the solution type back to HFSS without hybrid regions and set up the volume with FEM only. In one hour, the adaptive meshing process did not complete, and at the time of the last iteration, the mesh had 21 million elements. It is safe to say that this simulation would have taken more than a day to solve on the same hardware, showing that using the hybrid solve represents at least a 300% speedup in solve time for our case.

 

Figure 8: Comparison of FEM only and Hybrid matrix sizes

 

Let’s take a look at how our base case performs. As expected for an electrically large dish antenna, the 3D pattern is very directive, as plotted below for the 20 GHz case. The maximum gain is 37.5 dB at the anticipated theta = 90 degrees, phi = 90 degrees.

Fig 9: 3D radiation pattern for Antenna Toolkit dish

 

For the case of spontaneous feline loading, a CAD model of a sleeping cat was downloaded from a 3D file hosting site and assigned the appropriate material properties: a relative permittivity of 50, and a bulk conductivity of 0.2, as found corresponding to aggregate human electrical properties.

Fig 10: Material properties and 3D model of sleeping cat

 

Now, since our simulation geometry has changed, we can re-evaluate whether PO is the proper solver for our case. Since it may be the case that a ray could bounce off our catnapping pet and onto the reflector, the most appropriate solver would now be SBR+ as per our flowchart in Figure 5. The reflector was re-assigned to be an SBR+ region as described in Figure 7, and likewise, the cat was assigned as an SBR+ region.

The inclusion of our cat resulted in an increased solution time of 21 minutes and would compare similarly to an equivalent FEM simulation as the base case seen in Figure 8.

Now that we have both sets of results, we can see just how badly our antenna’s performance is affected. Plotted below is the total gain along the plane of symmetry (YZ) of our dish, for 3 points in the frequency range of interest, 10, 20, and 30 GHz. We see again a very directive radiation pattern for the un-loaded dish, with the main lobe having a maximum gain of about 37 dB for the 20 and 30 GHz case, and 34 dB for the 10 GHz case. Our kitten degrades the directivity of our antenna, with the maximum gain being reduced to around 28 dB for the 10 and 20 GHz cases, and 31 for the 30 GHz, making for an average 7 dB decrease in gain across the band.

Fig 11: Comparisons of gain patterns between feline-loaded and unloaded conditions. In bottom plots, dotted lines represent variation with cat.

 

The impact of an average 7dB drop in antenna gain over a band on the bandwidth of a satellite internet connection would be dependent on the coding scheme and the amount of margin used in the antenna. It would correspond to a reduction in signal strength to one-quarter of the cat-free baseline. It can be confidently concluded, however, that spontaneous feline loading is detrimental to antenna performance.

Solver Name	FE-BI (Finite Element Boundary Integral)	IE (Integral Equation)	PO (Physical Optics)	SBR+ (Shooting and Bouncing Rays +)	Dielectric Cavity
Description	Region assigned to the boundary between an FEM region and an unbounded solver in a hybrid solver	Solves Integral formulations of Maxwell’s Equations on boundaries	Ray Tracing simulation, capable of simulating one bounce	Ray Tracing simulation, capable of simulating many bounces	Uses FE-BI behind the scenes to avoid meshing  a large dielectric cavity
When to use?	(Implicitly applied to the boundary of any FEM region in a hybrid solve. )	Any design < 30*λmin ­	Simple designs > 30*λmin	Complex designs > 30*λmin	Large dielectric regions WITHIN FEM region
Material constraints	Only on boundaries	Dielectric or conducting objects or sheets	Dielectric or conducting objects or sheets	Dielectric or conducting objects or sheets	Uniform dielectric cavities
Example Applications	-Multiple antennas to be solved by FEM separated by a large distance	-Antenna placement, co-site - Radar Cross-Section (RCS) of smaller objects	-Large Single Reflector Antennas -Antenna placement on platform -RCS of large convex objects	-Large Multi-reflector antennas -Antenna placement of many antennas -RCS of large objects of arbitrary geometry	- Large lens for loading

Appendix 1: Relative comparison of hybrid solvers

 

 

 

 

Parallel Scalability – It’s… Complicated

Hello, today we’re going to talk about the theory of numerical method parallelization for scalable high-performance computing simulation, using a simple computational fluid dynamics case. Did your eyes just gloss over reading that? Well, don’t worry! This is a practical discussion intended for users to find real-world benefits to speed up their computations, not some deep dive into mathematics, and I’ll even bring along a few analogies to help it all make sense.

First, the background and some definitions.

“Numerical methods” is a broad term for the field of mathematics by which the solutions to equations are found through developing algorithms and offloading the extensive calculations to computers. It turns out, certain mathematical equations are just simply too complex for humans to solve by hand, so we instead develop a roadmap of repetitive calculations for a computer to follow, and the computer then executes them many billions of times until the solution is obtained. A particular subset of these routines is called “iterative”. For these types of methods, a computer is instructed to “iterate” (or take incremental steps) as it “converges” toward (or slowly approaches) the solution. Since that solution is not typically known ahead of time (or why would we be doing the analysis?), a layman’s description of this could be “guess and check” – we start with a guess and some mechanism to judge error, and then methodically reduce the error in subsequent guesses until we determine the guesses are close enough.

By and large, most of the tools of simulation follow similar iterative approaches. The example we’re discussing today, that of Computational Fluid Dynamics, or CFD, is certainly one of the most widely known versions and if you’re reading this blog, I assume you’re familiar with the field. Here, we solve a series of equations, known as the Navier-Stokes equations through iterative methods. I won’t go into the mathematics here (I promised I wouldn’t), but the gist of it is this: they depict the motion and physical properties of fluid when the user gives given boundary conditions and geometry of a physical problem to solve.

The reason I’m reminding you of those equations for this discussion is that they are considered “highly coupled”, which means they must be solved simultaneously. You can’t know the density somewhere without knowing the pressure and temperature. You can’t know the pressure and temperature without knowing the velocity and streamline history, and you can’t determine that velocity field information without knowing the pressure field information, which relies on momentum (a product of density and velocity) and viscous forces (also a function of velocity gradients). Furthermore, the solution in one geometric space changes the result in other geometric spaces nearby, so you can’t parcel out the geometry and solve each portion independently.

All of this is what causes the field of parallel distribution to be so interesting. Let’s introduce one of those analogies here:

Imagine an exam given to middle school students learning multiplication. The exam has 60 questions on it – each one a simple product of two numbers (3*2=6, 5*7=35, etc.). Say the teacher gives the test to a student and records their time to completion. Each student would differ, certainly, but let’s assume this exam takes an average of 4 minutes (or 4 seconds per problem) for the students in this class. This type of computational method would be called “serial”, meaning each equation is solved sequentially until the exam is completed. Now, let’s instead assume there are 30 students in the classroom and the teacher changes the assignment, now giving each student 2 of the questions to solve. This would be called a “parallel” method of computation. The time savings would be hugely significant, resulting in the entire class finishing in the time it takes the slowest student to complete their 2 questions (say 12 seconds). Since there were 30 times as many minds being devoted to the problem, and the solution time was cut by a factor of 20 (12 seconds/4 minutes), we could say that the “parallel efficiency” was 20/30 or 66.7%. The reason why it isn’t 100% was that every student did not have an identical time to completion and therefore was not given an appropriate balance of tasks. In the field of numerical methods, this is called “load balancing” and is a huge field.

Now let’s introduce a new concept. Let’s say the students are arranged in a circle, and instead of both questions given to the student being independent, let’s say the second question each student was given relied on the answer to the first question of the person sitting to their left. Something like:

Student #4 -

1) 5*3 = __

2)  4*x = __ where x is the solution to question 1 from Student #3

Therefore, at the completion of question #1, every student is required to share their answer with the person to their right, while receiving the answer from the person on their left. They can then carry out solving question #2.  Remember when I mentioned how in fluid dynamics the solution to one location relies heavily on the solution to the other nearby locations?

This communication time certainly would impact the overall solution time, resulting in a further reduction in parallel efficiency. If those students were very quick to communicate, this could maybe only reduce it to 50% (a total exam time of 16 seconds), but it’s far more likely the students’ communication time would drop that number to 20% (a total exam time of 40 seconds).

So now you can see how two major factors influence the efficiency of parallel scalability

1) Load balancing to ensure each task is completed in as close to equal time as possible.

2) Communication speed, to ensure no process is sitting idle waiting for information to be conveyed.

These exact same concepts apply directly to computational parallelization. The students in this analogy are computer cores, and the communication is done through memory channels. And the types of problems being solved can motivate very different architectures. Graphics processing, for instance, which typically does not require much coupling to other core processes (i.e., the rendering of one section of the screen doesn’t typically require another part of the screen to be rendered first) can be done on hugely parallel arrays, which is why modern graphics cards have several thousand compute cores. Think of this as a class of 1000 kids each independently doing 1 question on a 1000 question exam. Other types of computations though, such as those found in CFD, require a significant amount of communication, so, therefore, can be quite slow in such highly parallel environments as each core is bogged down waiting for others to convey the information needed to proceed.

So now, let’s dive into our CFD example.

We’ll start with something that could not be simpler – the straight duct – and we’ll assign it simple dimensions to make meshing very simple (Figure 1).

 

Figure 1: Geometry used for analysis. Flow direction from bottom-left to top-right.

 

And to illustrate a point, I’m going to take things to extremes – I’ll use a structured grid ranging from extremely coarse to extremely fine. Somewhere in the middle will be a mesh-independent solution where the results can begin to be trusted for accuracy, but our goal here isn’t to find that, our goal is solely to assess the speed of the computation under a range of parallel conditions.

However, another interesting component is worth discussing here. To converge a solution in a reasonable time, you need to consider 1) how many iterations are required for convergence and 2) how fast each iteration computes. This post attempts to address point #2 with how computer architecture and core count results in different iteration speeds. But the size and structure of the grid and therefore how well it captures the necessary gradients of the flowfield can influence #1 by how many iterations are required for convergence. Assessing the entire time required would require fully converging the solution under each scenario and comparing the results – a much more rigorous exercise and of course completely unique to each problem type solved. The important (and potentially unintuitive) point here is that there may be times where a finer mesh converges in less time than a coarser mesh simply due to being more numerically stable and requiring fewer iterations.

For the mesh of this simple duct, we used a structured H-grid (Figure 2). Not the most efficient or effective for capturing near-wall gradients, but again, we aren’t looking for accuracy here, we’re looking for computational efficiency and stability, and perfectly orthogonal elements are ideal for load balancing.

 

Figure 2: Mesh structure and 3 sizes used for analysis

 

A little aside on “partitioning”. A “partition” is a segment of the problem that is divided up and given to each process to solve (think “questions” in our student exam analogy). People may think of these as abstract computational concepts buried within the operating system’s code, but in the field of CFD, they are easily viewable physical regions of the geometry. I mentioned that there’s a huge field of study related to this, but luckily for us, Fluent engineers have largely solved this problem for us decades ago, and a modern CFD user hardly needs to be aware of it. If you were curious, however, there’s a field variable in Fluent called “Active Cell Partition” that returns a numerical value for each cell, indicating which processor core it is being assigned to. Pretty handy and allows me to show the partitions within this duct under a range of core counts 1 to 256 (Figure 3).

 

Figure 3: Visualization of mesh partitions in Fluent

 

The physics being solved in this duct will be 1 m/s velocity fed in one end, and backpressure of 0 Pa gauge applied to the other end. The walls will be no-slip walls, thereby creating a pressure gradient as the viscous boundary layers develop. The fluid will be incompressible water and we’ll use the SST turbulence model. Very simple.

For comparison, for each core count tested, we will analyze the time required to solve 100 iterations. The computational hardware is a cluster of identical compute nodes, each with two Xeon Gold 6242 processors on it and each connected via an InfiniBand fabric. Therefore, any computation with 16 or fewer cores will typically be solved on a single CPU with information passed within the memory lanes of the chip. Any computation with 32 cores will be solved on a single compute node requiring information to be passed across the board between the 2 chips, and any solution over 32 cores is solved over multiple nodes with information passed through InfiniBand.

Now a disclaimer: This is most certainly a case where we must be careful not to over-scrutinize. There are thousands of variables at play here, and the constants in all of it that allow us to even begin to analyze trends are:

• Simple flow in a perfectly constant cross-section duct
• Incompressible flow with simple gradients
• Perfectly orthogonal mesh elements
• Identical computer hardware between tests (core frequency, memory lanes, client submission machines, etc.)
• Repeatable partitioning/load-balance methodology
• Consistent pressure-based method
• Consistent turbulence model
• A defined number of iterations

However, like many things in simulation – trends are king, and the lessons we can learn from this data are likely to inform us going forward.

So now for the results data and interpretation (Figure 4). First, looking directly at the time required to compute the 100 iterations on a log-log plot we see the obvious – that as we increase mesh count, computational times increase while increasing computational cores reduces time. You can also search horizontally along the chart to deduce that by multiplying node count by 8, you need around 16-32x as many cores to solve it in the same amount of time – another way to think about scalability.  The second plot, converting the data to the linear scalability metric, helps illustrate the trends well and now we can more easily see the effects of diminishing (and eventually negative) returns. On this simple log plot, each curve follows an S-shape. Initially, the scalability is quite good, in that going from 1 core to 2 can have linear or even super-linear time savings with more benefit seen with higher node counts. Then, that return tapers off and eventually flatlines, or with very low node code begins to slightly increase the time required to solve. This is the equivalent of students spending a proportionately high amount of their time communicating their answers, and relatively low amounts of time-solving the math problems. Conversely though, the higher amount of work each student has (i.e., more mesh nodes to compute the solution on), the further that scalability holds on before it also begins to fall.

Another way to process the data for interpretation is through this fairly made-up term of “Core Productivity” (Figure 5), which is the product of the outputs of the simulation (namely the total iterations multiplied by all mesh nodes) divided by the inputs (number of CPU core-seconds). If we make the assumption that the mathematical output rate of a CPU core is relatively constant between runs, we can infer this to be a sort of metric of how much of that output is directly being used to solve the problem.  A bit of hand-waving, but the results are interesting. It tells us that for low core counts, the CPU is more productive when the node count is kept low while that trend flips at higher core counts. It also implies an optimum location along each curve, which can be seen on the higher node count cases, while the two lower node count cases are likely too small to see one.

 

Figure 4: Time in seconds required to compute 100 iterations with various mesh node and CPU core counts. (Water in duct, SST)

 

Figure 5: Core Productivity metric for various node counts (Water in duct, SST)

 

From this very simple study, we can start to take away a few lessons through trend analysis, but any discussion of specific inflection point locations or rules of thumb can go right out the window as soon as any of that huge list of variables change – computer to computer, problem to problem, mesh to mesh. Some of the lessons that can be seen:

• Increasing core counts can have a positive effect on solving time, but it can’t be expected to maintain linear scaling forever. It will have diminishing returns. Sometimes the returns can be negative.
• The higher the mesh node count, the better the scalability of higher core counts will be – up to a point.
• For HPC environments with sufficient licenses to support it, the highest productivity will therefore often be to split the cluster up into separate jobs runs in parallel than to run each job sequentially with the maximum number of nodes.

Hopefully, this was a helpful discussion and lent some insight that can inform our intuition of how parallel scalability works in practice. It turns out there’s much more going on than just throwing computational power at the problem. It’s… complicated.