Nominet Maintenance Reminder

[webber]

So you're suggesting there is a trigger that will flag that the domain is about to drop at which point you run DAC at full speed.
If there is such a reliable trigger why not just fire a short burst of preemptive create commands?
Of course if the trigger is unreliable then it would make more sense to validate it with DAC.
So an unreliable trigger might be a sudden latency change in an EPP command for example.
<domain:info>, which is not subject to AUP, always returns the same result, but maybe it's the time it takes that is the trigger?
I might just run a test... how many such queries per seconds would you say you can run before you upset the folks at Nominet?

 

[ukbackorder]

*Exactly* my point. No DAC needed.
I don't think nominet would get upset at all if there's no breach of AUP going on. The EPP is already getting hit hundreds of times per second. Now you just need to figure out what they're looking for.

 

[webber]

When the DAC caching bug was in use, what kind of latency changes did you see then that would constitute a trigger?
I'm just trying to get an idea of what timing variations might happen...

 

[ukbackorder]

You have to test. There are multiple factors. Of course it might all be wrong. But ask yourself why people need more than one legitimate tag.. for what benefit. And why domains seem to be registering before the DAC updates. And why there are so many EPP commands being run. All points one way.

 

[super-whois]

This is just speculation, but whilst a domain is in the process of being deleted, then there will be a lock on the database for that entry, resulting in a noticeable delay to all queries for that name. For an error such as "the domain is not on your tag", then the database row needs to be checked that the TAG matches.

If this is the case, then I think Nominet could close this flaw quite quickly, by adding an index on the (domain, tag) pair.

 

[webber]

Some early results seem to be all over the place in my case: from 6 to 15ms, but mostly 7 or 8ms

 

[webber]

First, a word of caution:
I tried <host:create> on a dropping domain and after about 1000 of V293 Name server is invalid - parent domain does not exist, it turned into V299 You have exceeded usage limits for operations which fail due to the non-existence of superordinate domains. You will be blocked for 24 hours.

I ran these tests on dmc.uk from the same server, so timestamps are in sync.
These are just some quick test on a sample of ONE, so not sure any meaningful conclusions can be drawn.

The DAC was running at about 60ms interval:

23:42:09.002, dmc.uk,Y,N,2017-10-23,2020-10-23,123-REG
23:42:09.068, dmc.uk,Y,N,2017-10-23,2020-10-23,123-REG
23:42:09.133, dmc.uk,Y,N,2017-10-23,2020-10-23,123-REG
23:42:09.198, dmc.uk,Y,N,2017-10-23,2020-10-23,123-REG
23:42:09.263, dmc.uk,Y,N,2017-10-23,2020-10-23,123-REG
23:42:09.328, dmc.uk,Y,N,2017-10-23,2020-10-23,123-REG
23:42:09.393, dmc.uk,Y,N,2017-10-23,2020-10-23,123-REG
23:42:09.459, dmc.uk,Y,N,2017-10-23,2020-10-23,123-REG
23:42:09.524, dmc.uk,Y,N,2017-10-23,2020-10-23,123-REG
23:42:09.589, dmc.uk,N
23:42:09.590, Domain dmc.uk is available

In parallel I had an EPP connection running <domain:info> queries on the same domain every aprox 10ms which always returns
'V096 Domain name is not registered to your tag' no matter what domain you use, whether it is available, registered, or suspended, if it's not on your tag.

23:42:09.007, RTT: 0.007
23:42:09.019, RTT: 0.008
23:42:09.029, RTT: 0.008
23:42:09.038, RTT: 0.007
23:42:09.048, RTT: 0.008
23:42:09.059, RTT: 0.009
23:42:09.071, RTT: 0.010
23:42:09.081, RTT: 0.010
23:42:09.089, RTT: 0.008
23:42:09.099, RTT: 0.008
23:42:09.109, RTT: 0.008
23:42:09.120, RTT: 0.009
23:42:09.128, RTT: 0.007
23:42:09.140, RTT: 0.009
23:42:09.150, RTT: 0.009
23:42:09.161, RTT: 0.010
23:42:09.169, RTT: 0.008
23:42:09.179, RTT: 0.008
23:42:09.189, RTT: 0.008
23:42:09.200, RTT: 0.009
23:42:09.210, RTT: 0.009
23:42:09.220, RTT: 0.009
23:42:09.229, RTT: 0.008
23:42:09.239, RTT: 0.008
23:42:09.249, RTT: 0.008
23:42:09.259, RTT: 0.008
23:42:09.269, RTT: 0.008
23:42:09.278, RTT: 0.007
23:42:09.291, RTT: 0.010
23:42:09.300, RTT: 0.009
23:42:09.311, RTT: 0.009
23:42:09.318, RTT: 0.006
23:42:09.331, RTT: 0.010
23:42:09.339, RTT: 0.007
23:42:09.351, RTT: 0.009
23:42:09.362, RTT: 0.010
23:42:09.371, RTT: 0.009
23:42:09.380, RTT: 0.008
23:42:09.390, RTT: 0.008
23:42:09.399, RTT: 0.007
23:42:09.413, RTT: 0.010
23:42:09.424, RTT: 0.011
23:42:09.432, RTT: 0.007
23:42:09.442, RTT: 0.008
23:42:09.451, RTT: 0.007
23:42:09.463, RTT: 0.009
23:42:09.473, RTT: 0.009
23:42:09.483, RTT: 0.009
23:42:09.493, RTT: 0.009
23:42:09.503, RTT: 0.008
23:42:09.512, RTT: 0.008
23:42:09.521, RTT: 0.007
23:42:09.535, RTT: 0.010
23:42:09.542, RTT: 0.007
23:42:09.553, RTT: 0.008
23:42:09.565, RTT: 0.010
23:42:09.574, RTT: 0.009
23:42:09.583, RTT: 0.007
23:42:09.597, RTT: 0.012
23:42:09.619, RTT: 0.022
23:42:09.645, RTT: 0.026
23:42:09.658, RTT: 0.013
23:42:09.667, RTT: 0.009
23:42:09.675, RTT: 0.007
23:42:09.685, RTT: 0.007
23:42:09.699, RTT: 0.009
23:42:09.709, RTT: 0.009
23:42:09.717, RTT: 0.007
23:42:09.729, RTT: 0.009
23:42:09.739, RTT: 0.009
23:42:09.748, RTT: 0.008
23:42:09.757, RTT: 0.007
23:42:09.768, RTT: 0.008
23:42:09.778, RTT: 0.007
23:42:09.788, RTT: 0.008
23:42:09.797, RTT: 0.007

And another EPP connection running <domain:delete> at same interval which always returns V096

23:42:09.005, RTT: 0.006
23:42:09.017, RTT: 0.006
23:42:09.027, RTT: 0.006
23:42:09.036, RTT: 0.006
23:42:09.048, RTT: 0.008
23:42:09.057, RTT: 0.007
23:42:09.068, RTT: 0.007
23:42:09.080, RTT: 0.009
23:42:09.087, RTT: 0.006
23:42:09.098, RTT: 0.007
23:42:09.107, RTT: 0.006
23:42:09.118, RTT: 0.006
23:42:09.128, RTT: 0.006
23:42:09.137, RTT: 0.006
23:42:09.148, RTT: 0.007
23:42:09.159, RTT: 0.008
23:42:09.168, RTT: 0.007
23:42:09.176, RTT: 0.005
23:42:09.187, RTT: 0.006
23:42:09.198, RTT: 0.006
23:42:09.207, RTT: 0.006
23:42:09.218, RTT: 0.006
23:42:09.227, RTT: 0.006
23:42:09.238, RTT: 0.006
23:42:09.248, RTT: 0.007
23:42:09.258, RTT: 0.006
23:42:09.268, RTT: 0.007
23:42:09.280, RTT: 0.009
23:42:09.288, RTT: 0.007
23:42:09.299, RTT: 0.008
23:42:09.309, RTT: 0.007
23:42:09.322, RTT: 0.010
23:42:09.329, RTT: 0.007
23:42:09.338, RTT: 0.006
23:42:09.351, RTT: 0.009
23:42:09.359, RTT: 0.007
23:42:09.369, RTT: 0.007
23:42:09.382, RTT: 0.010
23:42:09.389, RTT: 0.007
23:42:09.403, RTT: 0.010
23:42:09.411, RTT: 0.008
23:42:09.422, RTT: 0.009
23:42:09.429, RTT: 0.006
23:42:09.441, RTT: 0.006
23:42:09.450, RTT: 0.006
23:42:09.461, RTT: 0.006
23:42:09.472, RTT: 0.008
23:42:09.482, RTT: 0.008
23:42:09.490, RTT: 0.006
23:42:09.502, RTT: 0.007
23:42:09.512, RTT: 0.008
23:42:09.519, RTT: 0.005
23:42:09.532, RTT: 0.007
23:42:09.540, RTT: 0.006
23:42:09.551, RTT: 0.006
23:42:09.564, RTT: 0.009
23:42:09.572, RTT: 0.007
23:42:09.581, RTT: 0.006
23:42:09.597, RTT: 0.012
23:42:09.624, RTT: 0.027
23:42:09.648, RTT: 0.024
23:42:09.660, RTT: 0.012
23:42:09.665, RTT: 0.006
23:42:09.675, RTT: 0.005
23:42:09.685, RTT: 0.005
23:42:09.696, RTT: 0.006
23:42:09.706, RTT: 0.006
23:42:09.716, RTT: 0.006
23:42:09.726, RTT: 0.006
23:42:09.738, RTT: 0.008
23:42:09.747, RTT: 0.007
23:42:09.756, RTT: 0.006
23:42:09.766, RTT: 0.006
23:42:09.776, RTT: 0.006
23:42:09.786, RTT: 0.006
23:42:09.796, RTT: 0.005

There are very clear spike in latency both with <domain:info> and <domain:delete>, but they happened at about 23:42:09.600 which is at least 10ms after the domain became available according to the DAC

I checked a few seconds before as well but couldn't notice any obvious spikes

Any thoughts?

 

[super-whois]

which is at least 10ms after the domain became available according to the DAC

Your EPP times appear to be received times, could the DAC times be sent times?

 

[timter51]

The EPP slowdown could be everyone still using DAC (which is a lot) sending off all their EPP create commands.

 

[webber]

Your EPP times appear to be received times, could the DAC times be sent times?

All timestamps, including DAC, are received time. My DAC RTT is very consistent at 3ms

The EPP slowdown could be everyone still using DAC (which is a lot) sending off all their EPP create commands.

Indeed, the epp slowdown corresponds to a spike in <svTRID> counter from an almost clockwork 4 per each 10ms, to over 40. But even though the high number of EPP requests is sustained for almost 100ms the EPP latency recovers quite fast.

23:42:09.017 0
23:42:09.027 4
23:42:09.036 5
23:42:09.048 3
23:42:09.057 5
23:42:09.068 5
23:42:09.080 3
23:42:09.087 7
23:42:09.098 3
23:42:09.107 4
23:42:09.118 3
23:42:09.128 3
23:42:09.137 4
23:42:09.148 4
23:42:09.159 3
23:42:09.168 6
23:42:09.176 4
23:42:09.187 3
23:42:09.198 4
23:42:09.207 4
23:42:09.217 8
23:42:09.227 3
23:42:09.238 3
23:42:09.248 4
23:42:09.258 4
23:42:09.268 3
23:42:09.280 4
23:42:09.288 4
23:42:09.299 4
23:42:09.309 3
23:42:09.322 4
23:42:09.329 5
23:42:09.338 4
23:42:09.351 4
23:42:09.359 2
23:42:09.369 4
23:42:09.382 3
23:42:09.389 4
23:42:09.403 3
23:42:09.411 3
23:42:09.422 4
23:42:09.429 5
23:42:09.441 5
23:42:09.450 5
23:42:09.461 4
23:42:09.472 3
23:42:09.482 6
23:42:09.490 3
23:42:09.502 4
23:42:09.512 3
23:42:09.519 4
23:42:09.532 3
23:42:09.540 5
23:42:09.551 4
23:42:09.564 6
23:42:09.571 4
23:42:09.581 4
23:42:09.597 5
23:42:09.624 40
23:42:09.648 41
23:42:09.660 32
23:42:09.665 26
23:42:09.675 25
23:42:09.685 19
23:42:09.696 8
23:42:09.706 18
23:42:09.716 7
23:42:09.726 20
23:42:09.738 3
23:42:09.747 9
23:42:09.756 3
23:42:09.766 4
23:42:09.776 6
23:42:09.786 4
23:42:09.796 4

 

[cyberdomains]

@webber nice work.

Are you sure those EPP times are correct? My DAC RTT is 1.8ms, but I only manage 48ms for the domain:info command. I've tried from two different data centres. EPP hello command also measures the same.

 

[super-whois]

but I only manage 48ms for the domain:info command. I've tried from two different data centres. EPP hello command also measures the same.

Is that for the command only, or including the login on a new connection? 48ms seems rather high!

 

[cyberdomains]

Just for the command! Tbh I just always assumed EPP response times were quite high. I've been concentrating on DAC too much I think

 

[ukbackorder]

Nominet added an artificial delay like the time delay dac. I heard they also added delays to the whois and whois2. This suggests they don't know what's going on but don't think it should be running the way it is.

 

[webber]

Are you sure those EPP times are correct? My DAC RTT is 1.8ms, but I only manage 48ms for the domain:info command. I've tried from two different data centres. EPP hello command also measures the same.

I never explicitly measured RTT for DAC before this, but looking at logs from past days I see the timestamps between <greeting> <login> and <hello/> are 5 to 10 ms apart. Since you have a better DAC RTT your problem is not the route latency but probably the way you read and write to the socket.

Nominet added an artificial delay like the time delay dac. I heard they also added delays to the whois and whois2. This suggests they don't know what's going on but don't think it should be running the way it is.

Well that's shocking. How can you have full control and full logs and not work out what going on!? Weren't they supposed to be specialists in something... I can't remember what it was... cyber-something? Ah, yes, cybersecurity! Anybody knows what that means cause Nominet don't seem to know?

 

[lazarus]

One way to fix it would be to for nominet to aggrigate the create commands (1 per TAG) then randomise them within 200ms blocks.
This would resolve everything outside of multi tagging.

Job done

 

[webber]

One way to fix it would be to for nominet to aggrigate the create commands (1 per TAG) then randomise them within 200ms blocks.

I was thinking something similar as a solution but you risk legally running a lottery which you would need a licence for

 

[lazarus]

I was thinking something similar as a solution but you risk legally running a lottery which you would need a licence for

Not sure of the legalities but I was thinking more of a raffle which is probably covered under lottery rules?

 

[lazarus]

Whatever happens. Nominets new system is probably not going to block the current exploit. It is probably something to do with DNS. Randomisation of create requests is as far as I can see, the only way to truly stop all past and future exploits.

 

[webber]

It is probably something to do with DNS.

From the DNS perspective a suspended domain does not exist.
dig @nsa.nic.uk bix.co.uk returns no NS
Maybe it returns NS just before it drops?

I just tried updating one of my domains and there was a 30+ seconds delay in seeing that through dig