I recently came across something that was not available via EC2 API – lists of IP address ranges for each EC2 region. AWS team maintain such lists in their forums – currently at http://developer.amazonwebservices.com/connect/ann.jspa?annID=735 (I say “currently” because annID has already changed at least once). I asked for API, but in the meantime since I needed this now, I wrote a simple python script to scrape and parse the data – http://gist.github.com/559397 or embedded below. Enjoy! By the way, who would have thought that one would need to resort to scraping AWS, the very pioneers of infrastructure APIs, to get this simple bit of information?

I think that the biggest challenge for Infrastructure as Code today is its current lack of anything resembling a Makefile – a relatively simple description of how input could be transformed into output ready for use end to end, given a set of basic tools or a preset build environment (for example, for a project written in C it would be apt-get install build-essential on Debian and its derivatives). If you want an example, please take a look at deployment instructions for openstack/nova (”on the cloud controller, do this… on volume node, do that…”). While it is indeed infrastructure code, its end-to-end build and deployment instructions are provided in textual form, not as code.

Why is it a problem you may ask. First and foremost, build/deploy instructions provided in textual form can’t be easily consumed by a machine – it feels like we are back in the dark ages, without APIs where all work must be performed manually.

Secondly, because they are not fully formalized, they can’t be as easily shared – there could be many uncaptured context requirements that could lead to different people transforming identical inputs to outputs that would not function identically. And if they are not shared, same functionality is being worked on by many separate teams at the same time, which leads to incompatible, sometimes competing implementations and creates wasted effort by not allowing code reuse.

Thirdly, since they are not code, they are not as easy to test and verify test coverage for, or to fork and merge, or to port to other platforms.

My point is that while individual parts or steps of an infrastructure deployment could be automated, a whole thing rarely is, especially when a system is to be deployed to multiple hosts connected over the network. This would be similar to a software project with various directories, each with its own Makefile but without a top-level Makefile – such that you’d have to follow a HOWTO telling you which arguments to pass to make in each directory and in which order to run the commands.

What to do? I call on all infrastructure projects to make every attempt to ship deployment instructions not as textual step-by-step howto documents, but as code – be it Chef cookbooks, Puppet recipes, shell scripts, Fabric/Capistrano scripts and so on, or a combination of any of the above. Please consider providing cloud images (in at least one region of at least one public cloud) with your canonical build environment (your equivalent of build-essential). Please consider including canonical network topologies for your deployment – since you can’t predict IP addresses each user is going to allocate, all configuration files will need to be autogenerated or built from templates.

I am well aware it’s easier said than done, but if we do this, I hope a tentative consensus on best practices for infrastructure as code deployments could emerge over time which could then facilitate creation of a common “infrastructure make” tool.

If I were to query each endpoint one after another, I quickly discovered it would take too long. Therefore, I created a small helper class called BotoWorkerPool (in lib/boto_worker_pool.py), which wraps Python’s standard threading module around calls to boto – this helps achieve some amount of parallelism without introducing significant complexity of dealing with and sharing data among multiple processes. This also allows to potentially migrate to processing or multiprocessing libraries in the future, which offer threading-like interfaces for a multi-process model.

There are 2 tools at the moment.

onesnapshot.py creates new snapshots for all volumes that already have one snapshot marked with “__onesnapshot__” token. The rationale for this tool came in part from the following statement on AWS main page for EBS about durability:

The durability of your volume depends both on the size of your volume and the percentage of the data that has changed since your last snapshot.

imageequiv.py takes AMI ID, kernel ID or ramdisk ID and finds equivalent IDs in all regions, based on matching name or manifest file location. This tool is a response to the following tweet of mine:

wanted – equivalence lists for kernel and ramdisk images (aki-, ari-) across all ec2 regions

Hope these are useful to someone.

Suppose there is a town with just one public IaaS cloud provider, and that every business in the town runs their own IT: some by hosting it on-premises, some by running it in the cloud. It seems reasonable to imagine that the cloud provider obeys the following rule: it runs IT for all and only those businesses that do not run their IT on-premises.

Under this scenario, we can ask the following question: is cloud provider’s own IT in the cloud or on-premises? (remember that the cloud provider itself is also a business)

Of course this “faux” paradox is not a paradox at all. As I suggested on Twitter, “if you build a cloud, to your customers it will indeed look like a cloud; but to you it will look like a regular datacenter.”

Availability of your system, just as any other part of your service, is a feature. Just as with anything else, one needs to invest time, effort and resources in building it out. And whatever you dedicate to availability (such as development time) can’t be used for other features – this is what’s known as opportunity cost. If you could put same resources to a better use somewhere else, investing them in availability may not be the optimal decision. Additionally, availability draws from your complexity budget which is going to impact other areas – HA systems tend to be more complex and hence require more effort to develop, maintain and improve them over time. Availability, just as any other feature, has a price tag that you will have to pay to get it. Because you own your site’s availability, it’s up to you to decide how much availability you want AND can afford to build.

The last point is very important. Our daily lives are filled with points of failure – home appliances (can break), a usual route you take to work (could be impacted by road construction), your regular coffee place (your favorite barista could transfer to a different location). Do you maintain 2 different non-overlapping routes to work? Or do you frequent 2 coffee shops in order to have an alternative if one shop drops from your list? In other words, in our lives we regularly forgo availability when it doesn’t make sense – why shouldn’t we follow the same rule in our professional lives?

Availability is not a binary option. You could have all-active N-tuple, you could have active-active pair, you could have an active-passive pair with automatic failover, or same active-passive pair with manual failover. And finally, in today’s cloudy world, you could also have just a single resource with ability to replace this resource quickly if it goes down. Options include geographic redundancy, vendor/provider diversity, and so on. Availability could be as simple as host your systems at a very reliable provider. Or at the very least – be able to detect when there is a problem and be able to restore the system within a preset amount of time. Different levels of availability obviously don’t cost the same – pick one that you want and can afford.

Secondly, if your overall service consists of multiple smaller parts, you are free to choose different levels of availability for individual parts. Anything which responds to synchronous calls (a call that expects a reply immediately) – like web front door – may have one level of availability (higher), background jobs may have lower level. Designing each subsystem with appropriate level of availability will reduce your costs and most likely will let you save some of complexity budget for other things.

Thirdly, while availability is a single metric, problems that impact it are not. Some problems could be frequent and easy to deal with, other problems could be rare and catastrophic. Do you want to build your service to withstand a failure of a host, all hosts, all of your ISP, entire Internet? It’s all about the tradeoffs between costs, severity of each type of problem and probability of these problems to occur.

Fourthly, remember that availability measures that you build are your defenses against problems. A particular type of problem that you want to protect against, requires an availability measure targeted at this very problem – matching it by functionality, size and cost (a single defense measure may work against multiple threats). Imbalance in any of these three categories between your defenses and the problems they are meant to prevent will lead to suboptimal results. After all, you don’t use a shield to defend against a cannon and you don’t duplicate your entire operation into the second datacenter just to protect against a router failure.

And finally, beware of peer pressure. If your web front door’s availability costs $1m per month and it’s bringing in $10m per month worth of revenues,  it can be a no-brainer. But if you are investing 50% of your complexity budget in availability just because everybody else is doing it, I think it could be a problem.

Going back to AWS and putting my amateur behavioral economist’s hat on, I am curious how many people decided to take advantage of lower price for lower availability of RRS. And even more interestingly, if S3 initially were at RRS availability and AWS announced better availability for higher price, would we end up with the same distribution of people using higher and lower availability?

It has been about 6 months since I last blogged about work, so I figured an update may be in order, especially since today CohesiveFT announced availability of VPN-Cubed on Flexiant’s cloud offerings.

We’ve been very busy on VPN-Cubed engineering side. Along with features already on the roadmap, we delivered several enhancements that were directly driven or requested by VPN-Cubed users. On the product support side, we continued to expand a range of devices with which VPN-Cubed can do IPsec interop, which now include even ones I personally have never heard about before. We grew our experience and expertise in the area of troubleshooting intra-cloud and cloud-to-datacenter connectivity issues (there are many!). We’ve also worked on a few projects that required non-trivial topologies or interconnects, successfully mapping customer requirements to VPN-Cubed capabilities.

One theme that I have had in my head for some time now, is VPN-Cubed as the networking fabric of the Intercloud. Let me explain.

VPS was a predecessor of modern IaaS clouds. In VPS land, boxes are usually provisioned individually, one by one. Typical setups in VPS consisted of 1, 2 or 3 boxes. Networking 3 independent boxes together is relatively straightforward.

At the beginning of IaaS era, I imagine most setups were also 1 or 2 boxes. But as IaaS is gaining ground, topologies headed to the cloud are getting bigger, more complex and more dependent on access to external resources. Setting up networking consistently is becoming a bigger deal. But it’s not the end.

One of the roles of Intercloud is providing customers with an alternative (competition, in other words) – if one doesn’t like cloud A, she may take entire topology to cloud B. I’d say 99 of 100 public cloud justification documents being submitted to CIOs worldwide today include a statement saying something like this: “If this cloud provider fails to deliver what we need at a level we need it, we will switch to another provider.” This is actually not as easy in practice as it may sound.

Each cloud’s networking has unique aspects, no two are alike. Public IPs, private IPs, dynamic or not, customer assignable or not, eth0 private or public, cloud-provided firewall exists or not, peculiarities of firewall – these are some of the differences (as of today, I have set up boxes in 6 IaaS clouds with admin endpoints facing public Internet – I have seen many network setups). Taking images of N boxes from one cloud and dropping them in another cloud is well understood, recreating one cloud’s networking in another cloud is where the challenge is.

It is here where I think VPN-Cubed shines as a customer-controlled network abstraction – it’s an overlay built on top of service provider’s network, which allows it to be identical no matter what the underlying infrastructure looks like.

Same story plays out when an application is hyper-distributed and runs in multiple clouds or multiple regions of one cloud (where regions are connected via public Internet). And here as well VPN-Cubed provides an abstraction that allows one to treat all compute resources as being on the same network, regardless where they are actually located at the moment.

At the same time, VPN-Cubed can be appealing to topologies that don’t care about Intercloud. Networking and network security are areas that don’t get enough attention from cloud developers today, because developers are used to working within a perimeter. Excessively wide-open security group setups, using public IPs instead of private for communications, disabled local firewalls – these are all time bombs. They don’t affect the app right now (”look, it works!”) but they can be catastrophic over time when they could become an attack vector. For such topologies, VPN-Cubed provides a virtual perimeter that confines authorized communications to a mutually-authenticated tunnel encrypted end-to-end (are you sure you want to continue forcing your incoming web traffic to HTTPS but not encrypting writes and reads from app servers to database? or do you think application-level encryption could be better, faster or easier to maintain than transport-level?)

To get started with VPN-Cubed, visit http://cohesiveft.com/vpncubed. If you have a question about how VPN-Cubed can help in your particular use case, you can ask here.

UPDATE 08/21/2010: Randy Bias in his excellent new post touches on the very similar theme:

Where is the lock-in then? If it’s not the hypervisor, what makes moving from one cloud to another so difficult? Simply put, it’s architectural differences. Every cloud chooses to do storage and networking differently.

Devops as a way to promote “collaborative and communicative culture” (see John Allspaw’s comment) – “devops the culture” henceforth – was born out of frustration on both sides of the house (dev and ops) when it was time to push out and troubleshoot a new code release. In most web companies, a new release would happen on a weekly, bi-weekly or monthly basis – and the night of bad blood, fingerpointing, cursing, non-stop conference calls and IM conversations, all-nighters and so on would take center stage, with more of the same a couple of days later at a postmortem. Most frustratingly, anyone who has ever worked through anything similar inevitably realizes that the amount of stress could be considerably reduced by simple coordination beforehand when concerns could be raised and addressed.

Similarly, if you ever had to troubleshoot someone else’s code in a stressful situation, you would often realize that code in question was not written with such situations in mind – inconsistent and/or incomplete logging, lack of clear understanding of dependencies, failure to properly catch network socket exceptions. Again, this experience can be made much much better by close involvement of ops in parts of development process.

I believe it’s important to realize that “ops” in “devops the culture” does not stand for all of operations (which includes systems engineering, networking, storage and security). I specifically mean a subgroup of operations that most commonly is referred to as “application integration,” “application support” or “application engineering.” These are the folks who run the software infrastructure on top of which developers’ code is running. In bigger companies, these would be dedicated people. In smaller ones, the same people may also wear other ops hats such as networking or storage.

Now, don’t get me wrong – I would LOVE to see “devops the culture” applied to all silos in IT. But it’s easier said than done, at least for purely technical decisions. I have been a part of multi-silo technical teams tasked with making specific decisions. No matter how much discussion or coordination occurs, the decision still ends up being made by whoever knows the area best (have you ever witnessed developers giving suggestions to network engineers how to design the network? what about storage people advising security? and the best part – systems engineers advising developers how their code should run?). Hard to say it’s a bad thing either, especially from responsibility and accountability standpoint.

Hence, in my humble opinion, “devops the culture” is not about how to bring collaboration to all of IT. It’s about avoiding frustrating experiences related to running one’s own code in production.

Network, security as well as systems do play a big role in “devops the infrastructure as code” (see Adam Jacob’s comment) however – there’s no question about that in my mind. How well these areas will lend themselves to automation still remains to be seen however.

First of all, it’s worth noting that incentives are misaligned here. For publishers, cross-posting of public updates may make some sense – the more channels their message appears in, presumably the better. For readers however, the opposite is true – the more times you have to read the same update, the worse. As readers, we sometimes may want to engage in a conversation – and it’s far from clear how one should do it if you only receive a cross-posted update, not the original update on the network on which it was originally published.

Secondly, this emphasizes our current service-centric model of social media, as opposed to person-centric as it should be. In person-centric model, each individual will probably have several public streams (most will have only one I guess) and several private streams (family, friends, work). You let others subscribe to an appropriate stream (with or without authorization, depending on your settings) and they configure their reading service to filter your updates in or out. For example, I may want to follow someone’s public stream but would like to filter out their Foursquare notifications (unless they have a payload which says something more than “I am at X”). For more on this, please check out Blaine Cook’s recent post about identity.

And thirdly, this brings up a question of efficiency. Reading same tweets twice is inefficient no matter how you look at it. Un-connecting from someone on LinkedIn just because I don’t want to see their updates from Twitter is also inefficient. Unfollowing them on Twitter and reading their updates on LinkedIn may make sense, but then I don’t log in to my LinkedIn very often – and this ends up being inefficient too.

It’s interesting to note that the question of efficiency does not exist in the context of a single network. You either follow/friend an individual on a given network and receive their updates, or you don’t. This is a binary choice, which is relatively easy. It’s when the same updates get duplicated into multiple networks that the issue of efficiency is starting to play a role.

Efficiency is a tricky concept – it depends on the angle from which you are analyzing current situation. Therefore, in economics there is a concept of Pareto efficiency. An outcome is considered Pareto efficient if no one can be made better off without making someone worse off. Pareto-efficient outcomes have two interesting properties that apply to social media as well.

1. There could be multiple Pareto-efficient outcomes. Imagine you and I are walking on the street and see $100 in $10 bills on the floor. We could split it 60-40 or we could split it 70-30 or we could split it 50-50. Each of these outcomes is Pareto efficient by definition. Similarly, I can follow you on Foursquare and get your location updates. Or I can follow you on Twitter and you could configure your Foursquare to cross-post to Twitter, and I will get the same updates.
2. The fact that an outcome is Pareto efficient doesn’t necessarily mean it’s socially desirable or “the best” (consider your feelings if we split $100 above $70 for me and $30 to you).

We have yet to define “better off” in the context of social media. I can think of at least 2 forms here. The first, weaker, form could be “not miss an update in which one is interested in.” The second, stronger, form could be “weaker form + avoid receiving updates in which one is not interested in.”

Based on all this, it looks to me like what we have now is vendors racing to a weaker-form Pareto-efficient outcome – every service wants to be your destination for writing status updates and for reading those of your friends, without any regard for you, your wasted time re-reading same things many times, or the fact that most of users would actually prefer a stronger form. If such an outcome is achieved, it may be very difficult to change (because someone will have to be made worse off, by definition) if we want to get to a person-centric model.

What I would like to see instead of cross-posting (which essentially is cross-pushing of the same update to multiple channels) is cross-pulling and filtering – essentially creating customized activity streams based on reader’s preferences, not those of publisher. I am concerned however about the present lack of incentives for this behavior to emerge.

Query API vs REST API

To start, one should read this blog post by Jan-Philip Gehrcke about various types of AWS APIs and differences between RESTful and query API, and this blog post by William Vambenepe where he analyzes various IaaS API implementations (it’s a series of 3 posts). Then read description of Richardson Maturity Model by Martin Fowler.

In a nutshell, I think from practical standpoint, if one’s domain maps easily to a set of entities (nouns) and API operations on these entities are primarily CRUD, in this case one’s best bet is to go with at least Level 2 REST. If either doesn’t work, I’d go with Level 0 REST, which is essentially what query API is.

My main reason for not going with Level 0 when entities and operations do map, is that I hate to see this meta data go to waste because it doesn’t cost almost anything to include.

Between Level 2 REST and Level 3 REST, I think Level 2 is more practical. According to Fowler, “Level 3 introduces discoverability, providing a way of making a protocol more self-documenting.” It’s certainly a nice feature but I am not sure this added benefit justifies extra development effort and slightly increased complexity (some might argue it may actually reduce complexity though).

API frontend vs API methods implementation

Keep implementation of your API methods separate from whatever frontend you are deploying (REST, SOAP, etc). API methods are probably going to be the same no matter how they are called, so they should be frontend-independent. This will make it easier for you to introduce new frontends (AMQP, for example) and should facilitate code maintenance.

HTTP verbs

Read and delete operations are easy – they map to GET and DELETE.

Create and update are trickier. Canonical description of HTTP verbs can be found in Section 9 of RFC 2616 and I use the table here as an addendum. In short, for both create and update, if an operation is idempotent and URI of entity on which this operation is being performed is known, use PUT. Otherwise, use POST (it is often used on entities representing “factories” – say a factory of new postings; you don’t know URI of a posting before you create it, so you POST to a factory which will create a new entity at a new URI; note that POST is not idempotent).

Note the RFC definition of idempotent methods (9.1.2) – it’s not defined as “multiple invocations must lead to the same result as a single invocation.” It’s “(aside from error or expiration issues) the side-effects of N > 0 identical requests is the same as for a single request.”

HTTP return codes

Section 10 of RFC 2616 is a canonical description of HTTP status codes.

Successful completion should be signaled as HTTP 200 OK and, if it’s important for client to know that an entity was created as a part of operation, HTTP 201 Created. The latter may be redundant – code that handles 200 and 201 most likely will be identical or very similar.

Speaking of errors, I don’t think it’s practical to map each type of error to its own HTTP error code. Unexpected server side errors (frontend exceptions or uncaught exceptions raised by your API methods) could be HTTP 500 Internal Server Error. If a resource is not found, it should be HTTP 404 Not Found. If your API server uses an external service to perform certain operations and upstream service did not respond or returned an unknown error, I would signal this fact with HTTP 502 Bad Gateway.

The rest of the errors are all client-side, and I like to classify them into 2 categories. When something is wrong with submitted request (missing header, missing argument, argument of wrong type), I think server should return HTTP 400 Bad Request. This way server is telling the client that no matter how many times this request will be submitted, it won’t work and will produce identical response.

I then group all other client-side errors together and think they should lead to HTTP 403 Forbidden. It means request by itself is fine, but something is preventing server from executing it – such as a missing prerequisite. Re-submitting the request may work in this case, because by the time the request is re-submitted, something might have happened and prerequisite is already in place.

Error response could include application-level exception and its description – this way you are letting the client know exactly what was wrong. Whether processing these ends up automated or not – it’s up to the client.

Message formats

I can’t easily justify this one, but I feel that bodies of request and response should be in the same format (there could be exceptions – for example, when client must upload a binary artifact). vCloud does it this way – request body is XML, and response is XML. EC2 API sends request arguments in query string (because all requests are GET, since it’s query API) and response is XML. OCCI API defines request body as form-urlencoded (application/x-www-form-urlencoded) and response is XML as well (all of the above might support JSON as well).

I have 2 weak justifications for this.

Firstly, it somewhat mimics our regular human behavior. If 2 people are communicating in real time, they usually use same medium and same format. It’s rare when one person is on IM speaking English, while the other is on the phone speaking French – not saying it’s impossible but relatively rare.

Secondly, in the future I foresee a greater use of messaging in API operations (read this post by George Reese). Notions of request/response come from HTTP, in messaging it doesn’t matter – the same message could be response to one message and request to another. For example, a message requesting server start may lead to a message saying “server started” to the client. At the same time, the same “server started” message may go to an internal billing system, where it would be a request to start billing.

Having these message in the same format might be beneficial.

Command line tool

AWS set the bar with EC2 here. For every API call, they ship a command line tool to perform said call. No matter what you think whether it’s right or wrong, I think every provider should match this behavior. It’s a good practice after all – when someone is about to try API, it’s much easier to get going using command line tools instead of embedding API calls straight into the application.

Instead of EC2 practice of one command line tool per API call however (even though inside they still call ec2-cmd), I favor Sun Cloud’s approach – they were planning a single unified tool where an API call would be identified by an option or a subcommand.

Conclusion

As the Zen of Python goes, “[...] practicality beats purity.” This should be your main guiding principle when designing API server side.

On several occasions, when troubleshooting a connectivity issue or verifying a deployment’s security, I needed to check local and/or remote security group rules from within my instances. I never have AWS credentials on the instances, so I couldn’t use the API. And EC2 metadata server (169.254.169.254) returns only the names of security groups, not the rules themselves. In other words, I needed to answer the following question – “do remote instance’s security groups allow or not allow communications from this instance on TCP/UDP port X.” Note that trying to open a connection and not getting a response doesn’t answer this question – packets could be blocked by security groups or by remote instance’s local firewall (iptables). It turns out there is a way…

Let me first start with a warning. You probably are OK using this technique against your own instances, or instances of your friends, partners, vendors and customers with their permission and when you know specific ports you need to probe, for verification or troubleshooting purposes. Probing someone else’s instances or random ports may get you in trouble – such activities could fall into a category prohibited by AWS Acceptable Use Policy. You have been warned.

Summary

This technique works using private IP addresses, so both instances must be running within the same EC2 region.

To check if remote instance’s security groups allow communications from your instance on UDP port $X, do this:

traceroute -p $X -w 1 -q 1 -m 1 $REMOTE_INSTANCE_PRIVATE_IP

To check if remote instance’s security groups allow communications from your instance on TCP port $X, do this:

tcptracerute -w 1 -q 1 -m 1 $REMOTE_INSTANCE_PRIVATE_IP $X

(tcptraceroute is not a standard tool installed by default; you can get it from here or apt-get install tcptraceroute on Debian or Ubuntu)

If you see “1 {IP address of first hop} XX.XXX ms“, remote instance’s security groups allow such communications from your private IP address. If you see “1 *“, they don’t.

Background

This technique is based on one of the conclusions of my UDP hole punching post.

In it, it looked to me like each dom0 in the region knows about all security group rules in this region. Additionally, it looked like when dom0 knows that remote dom0 will not accept traffic, it doesn’t bother even sending it. If we assume they do it on dom0 with iptables, they probably simply DROP such traffic (at least I would if I were them). And if they do, they won’t send ICMP Time Exceeded packet back to us – and hence, traceroute won’t be able to report the first hop.

If we do get that ICMP packet from dom0 back, it means it didn’t drop our packet. Which means it knows the other dom0 currently has a rule in security groups allowing it.

I set max_ttl to 1, since first hop in traceroute is believed to represent dom0.

Obviously, I verified this theoretical hypothesis on a pair of EC2 instances in us-east-1, and didn’t see any indication that it could be wrong.

Protecting Against Such Probes

If you have instances in EC2, you may be wondering if you could somehow protect them from such probes. After all, AWS network monitoring is top-notch, but they have a lot of hosts to watch over, so some small amount of probing may still occur and you may not even know it.

I’ve got good and bad news on this front. The good news is that you can do it, the bad news is it’s going to be ugly. Essentially, to avoid disclosing the ports open in your security groups to such probes, you must configure your security groups not to allow connections from arbitrary instances in your EC2 region. In other words, you must exclude 10.0.0.0/8 from all IP-address-based rules (rules granting access by name of security group are not affected). It means that you have to replace each rule that references 0.0.0.0/0 with multiple rules individually referencing 128.0.0.0/1, 64.0.0.0/2, 32.0.0.0/3, 16.0.0.0/4, 1.0.0.0/5 (I would avoid 0.0.0.0 just in case – hence 1 as first octet), 12.0.0.0/6, 8.0.0.0/7 and 11.0.0.0/8. Yep, 8 rules instead of 1. You have to also repeat the same exercise for all IP subnets referenced in your rules that include 10.0.0.0/8.

Conclusion

In the future, of course, I hope security groups API could be extended to support functionality “allow connections from Internet excluding private IPs in this region”. I am going to add it to my previous wishlist.