In June of 2016 at the annual DAC conference, Teraproc previewed new license scheduling capabilities in OpenLava explaining how EDA licenses could be shared among different users, design teams and projects on the same cluster.
With the release of OpenLava 4.0, resource based pre-emption has arrived making OpenLava a much more compelling choice for EDA firms concerned about license management. New Functionality in OpenLava supports not only flexible license sharing on the same cluster, but license sharing across multiple clusters as well.
In this technical update to our earlier article on License Scheduling in OpenLava, we provide a closer look at License Scheduling in Teraproc OpenLava Enterprise Edition.
Please note: This is a copy of an article originally published on the Teraproc blog.
The need for speed in design environments
In the EDA design process, simulation is crucial. Given the high cost of device fabrication, firms need to perform extensive verification across many facets of the design using software from Cadence®, Mentor Graphics®, Synopsys® and others. Simulation can involve running literally millions of compute and I/O intensive jobs. To get these simulation jobs done in a reasonable amount of time, firms commonly use clusters of fast computers relying on schedulers like OpenLava to provide policy-based resource sharing among multiple teams and projects. Having faster processors, and more capacity for time and resource intensive tasks like verification and regression testing helps firms do more design iterations, reduce cycle times, and ultimately deliver a higher-quality design faster.
EDA Licenses – a critical resource
One of the key cost drivers in electronics design are EDA software tools. The tools used for device simulation are specialized, frequently costing more than the hardware they run on. As feature licenses are often in short supply, EDA managers need to ensure that licenses are allocated appropriately, and that usage is maximized.
License availability is often a limiting factor in simulations. For example, if a particular project should in theory have 30 licenses, but is only allocated 15, the simulation activity may take twice as long regardless of the amount of compute capacity available. This is because jobs will pend in queues waiting for licenses while processors sit idle. Inefficient allocation of licenses can impact deadlines, reduce the productivity of engineering staff and lead to underutilization of infrastructure.
Challenges with license management
A common approach to managing application licenses is to configure and track licenses as resources. Users then specify license resource requirements when submitting jobs. The scheduler will hold the job in a queue if the resource (license) is not available. This simple approach prevents a job from being dispatched from a queue only to fail at run-time when a licenses is not available.
In a large design environment, when multiple projects have different priorities, and the design work is performed across multiple sites, the above method does not work very well. Some challenges are:
- License checkout is first-come, first-serve. The workload scheduler prioritizes jobs and performs job preemption based on job slots, not external resources like licenses. If licenses are occupied by low priority jobs, high priority jobs have to wait.
- In an environment where multiple sites are sharing a license pool, there is no coordination between the schedulers at each location. When a license becomes available, multiple sites may detect this simultaneously and start to launch job thinking that they have exclusive access to the license. Only one job will be able to get the license, and the remaining jobs will fail.
To help customers overcome these challenges in license optimization, Teraproc has added new functionality in Teraproc OpenLava Enterprise Edition to support both license sharing within a single cluster as well as license sharing across clusters. We look at these two use cases below.
Use case 1: License sharing and preemption within a single cluster
To ensure that licenses are allocated to the highest priority jobs, OpenLava 4.0 Enterprise Edition supports resource based preemption. If an external resource is configured to represent a specific application license, when the resource is not available, and there is a high priority job waiting for it, the scheduler will try to find a low priority job that uses the resource and preempt it. A simple example that shows how license preemption works on a single cluster is provided below.
A license feature called “vcs” (representing Synopsys® VCS, a functional verification tool) is configured as a shared resource on the cluster. Also, five queues are configured with different priorities as shown below. These same queues will be used in the next use case as well.
$ bqueues QUEUE_NAME PRIO STATUS MAX JL/U JL/P JL/H NJOBS PEND RUN SUSP vcs_high 40 Open:Active 25 - - - 0 0 0 0 vcs_medium 30 Open:Active 25 - - - 0 0 0 0 vcs_low 20 Open:Active - - - - 0 0 0 0 vcs_reg 5 Open:Active - - - - 0 0 0 0 vcs_cmp 5 Open:Active 110 - - - 0 0 0 0
The QJobLimit can also be constrained for select queues. For example, we may want to make sure that no more than 25 jobs can run concurrently from the vcs_high and vcs_med queues to avoid all licenses being consumed by high priority jobs. OpenLava administrators can configure other policies at the queue level as well. For example, they may want to make sure that higher priority jobs dispatch to a host group containing the newest fastest servers to maximize performance and minimize license checkout time so that high priority users enjoy a better service level.
Running bhosts -s shows the status of the shared numeric resources representing licenses.
$ bhosts -s RESOURCE TOTAL RESERVED LOCATION vcs 60.0 0.0 prometeo ol1 ol2 ol3 ol4 ol5 ol6
We see there are 60 available vcs licenses. We then submit 60 jobs to the low priority queue to run sixty vcs simulations and consume all available licenses.
$ bsub -o /dev/null -R "rusage[vcs=1]" -q vcs_low vcstest1 Job is submitted to queue . $ bsub -o /dev/null -R "rusage[vcs=1]" -q vcs_low vcstest1 Job is submitted to queue . .. $ bsub -o /dev/null -R "rusage[vcs=1]" -q vcs_low vcstest1 Job is submitted to queue .
After submitting the jobs, we check the status of the jobs (using bjobs) and the status of our shared licenses resources (using bhosts -s).
$ bjobs; bhosts -s JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME 4728 david RUN queue_low prometeo ol1 vcstest1 Jan 21 15:20 4729 david RUN queue_low prometeo ol1 vcstest1 Jan 21 15:20 4730 david RUN queue_low prometeo ol1 vcstest1 Jan 21 15:20 4731 david RUN queue_low prometeo ol1 vcstest1 Jan 21 15:20 4732 david RUN queue_low prometeo ol1 vcstest1 Jan 21 15:20 4733 david RUN queue_low prometeo ol1 vcstest1 Jan 21 15:20 .. 4786 david RUN queue_low prometeo ol4 vcstest1 Jan 21 15:20 4787 david RUN queue_low prometeo ol4 vcstest1 Jan 21 15:20 RESOURCE TOTAL RESERVED LOCATION vcs 0.0 60.0 prometeo ol1 ol2 ol3 ol4 ol5 ol6
Note that all sixty licenses are reserved, one for each running job. We then submit a high priority job requesting one vcs license.
$ bsub -o /dev/null -R "rusage[vcs=1]" -q vcs_high vcstest2 Job is submitted to queue .
When this higher priority job needs a license, the most recently submitted low priority job is suspended (preempted) to release a vcs license. This allows the high priority job to claim the free vcs license and start. The preempted low priority job moves into SSUSP state, waiting for a free vcs license.
$ bjobs JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME 4788 david RUN queue_low prometeo ol4 vcstest2 Jan 21 15:20 4728 david RUN queue_low prometeo ol1 vcstest1 Jan 21 15:20 4729 david RUN queue_low prometeo ol1 vcstest1 Jan 21 15:20 4730 david RUN queue_low prometeo ol1 vcstest1 Jan 21 15:20 4731 david RUN queue_low prometeo ol1 vcstest1 Jan 21 15:20 4732 david RUN queue_low prometeo ol1 vcstest1 Jan 21 15:20 4733 david RUN queue_low prometeo ol1 vcstest1 Jan 21 15:20 .. 4786 david RUN queue_low prometeo ol4 vcstest1 Jan 21 15:20 4787 david USUSP queue_low prometeo ol4 vcstest1 Jan 21 15:20
Next, we display only suspended jobs using the bjobs -s command. We see the reason that the job cannot be scheduled. The resource requirement for a vcs license is not satisfied.
$ bjobs -s JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME 4738 david SSUSP queue_low prometeo ol1 vcstest1 Jan 21 15:36 Job's requirements for resource reservation not satisfied;
When the high priority job finishes, the vcs license is released, allowing the preempted low priority job to reclaim the license and start.
Resource preemption in OpenLava solves one of the major challenges with sharing licenses within a cluster, allowing sites to easily configure preemption policies based on resources.
Use case 2 – License allocation and preemption across multiple OpenLava clusters
In larger EDA environments, it is common to have different design teams working across different geographies. Depending on the tools vendor, licenses can often be shared among these different clusters. Ideally, cluster administrators want to make sure that expensive licenses are 100% utilized, but also that project team are getting their fair allocation of licenses.
Teraproc have developed a license broker for OpenLava Enterprise Edition that works across multiple OpenLava clusters to coordinate and schedule licenses from a shared license pool. An illustration of this use case is shown in Figure 2.
Flexible license sharing between sites
The scenario shown in Figure 2 comes from a real customer use case. For simplicity, we’ll focus on just one license feature as an example. This customer wants to share a pool of 150 licenses between clusters in five geographies. Statically allocating 30 licenses to each cluster would be fair, but would also be inefficient. We can imagine a scenario where users on Cluster 5 are very busy and there are no projects active on any of the other clusters. In this case 80% of our licenses would sit idle, while jobs on cluster 5 would be constrained because of a shortage of licenses.
Obviously we would like any cluster be able to borrow licenses from others, but have the ability for remote clusters to preempt workloads as intelligently when they have their own jobs to run and need to get their license resources back.
While the use case involves sharing licenses equally when there is contention (note the share=1 associated with each cluster in Figure 2), the license broker can incorporate different sharing ratios allowing sites with larger design teams or larger clusters to have a larger share of licenses by default. Also, minimum and maximum allocations per cluster can optionally be configured.
When the license broker is installed and configured, mbatchd on each OpenLava cluster contacts the license broker when there are jobs pending for licenses. The license broker then allocates licenses based on the configured policies for individual OpenLava clusters.
One of the most important features of the license broker is that it can work with all OpenLava mbatchd instances to preempt jobs (using the mechanism previously described in use case 1). This allows a high priority job on one cluster to preempt a low priority job on another cluster based on license requirements and availability.
How the license brokering algorithm works
When it comes to sharing licenses, the devil is in the details. The global license broker needs to liaise with multiple OpenLava schedulers, maintain an awareness of the workload on each cluster, and allocate licenses and preempt jobs as necessary to achieve a configurable sharing policy.
The flow charts in Figures 3a and 3b explain the license sharing policy in detail. The flow chart traces the logic from the perspective of a job being submitted on cluster 1, but the same logic is applied for jobs submitted on all clusters interacting with the global license broker.
In general, the algorithm will attempt to allocate any free licenses on the local cluster first. If free licenses are not available locally, it will then seek to allocate free licenses from the remote cluster with the largest available number of free licenses. If there are no free licenses available elsewhere, and the local cluster is already consuming its fair share of licenses, then the submitted job will simply need to wait because all licenses are in use and cluster 1 already has its fair share.
If cluster 1 does not have its fair share however, the license broker will need to free up a license on another cluster (this part of the process shown in Figure 3b). When it becomes necessary to preempt a license from another cluster, the license broker will target the remote cluster that is consuming more than its fair share. At this point it will start looking for jobs consuming the needed license feature in the lowest priority queue. If no jobs are found, it will continue looking in the next higher priority queue until it finds one or more jobs consuming the needed license. If there are multiple jobs dispatched from a queue with a particular priority level, the license broker will preempt the job that has been running for the shortest period of time to minimize wasted license and CPU resources.
If after running through all the different queues from low-priority to high-priority, the needed license features cannot be found on the cluster consuming the most licenses, the global license broker will then repeat the same process on the next cluster consuming the second highest number of total licenses.
Improved efficiency with license scheduling
Sharing licenses between sites using the license broker as describe above has multiple benefits:
- Organizations can avoid the need to purchase additional licenses to meet peak requirements at each location
- Cluster administrators can provide a predictable SLA that will help designers in each location ensure that a minimum number of licenses will always be available for planning purposes
- Engineering productivity is enhanced by allowing free licenses from other sites to be borrowed when available to increase capacity and reduce runtimes on other clusters
- Both licenses and computing assets are more highly utilized allowing firms to enjoy greater cost efficiency and a higher return on assets
- To learn more about license optimization including resource preemption features in OpenLava EE 4.0 and the Teraproc License Broker, contact Teraproc for a no-obligation consultation.