The things I have learnt whilst building a Drop Catch system

[gareth_brown]

Hopefully this is the right place to post this.

Why am I writing this post? I've been lurking around for a while and wanted to give something back, if I can.

Drop Catching is one of those things no one in the domain world wants to talk about. Well they won’t talk about their scripts! Why would they? It would be like Coke giving up their magic recipe.

So far it has been a 5 week project and I have lost count of how many hours it’s taken (and continue to take). It is still in development and currently off-line. Every day I tweak code, measure code speed and fine tune socket communication. It’s amazing how many hours you can spend measuring and analysing milliseconds. Okay, I am not going to give you any code but here are some things that I have learnt.

My software is currently written in C#, but I might move to C++ if it will provide faster timings. I am sure it won’t and hopefully below will explain why.
I will say up front, I do not know anyone here or anyone that Drop Catches, so I’ve been working in the dark. It’s been fun stumbling around trying to understand how it all works. To be honest it is surprising basic.

The process:

1. Repeatability query DAC for domain availability
2. If domain drops send EPP create request

My average timings:

These times can vary, not by a lot, but they do vary. The reason your timings will vary will be due to network loads at your data centre and those at Nominet. Nominet’s systems are going to be the slowest part of the system, so be careful what the time of day you test!

• Sending DAC query 0.0015ms
• Receiving DAC query 0.0020ms
• Sending EPP Request 0.05ms
• Receiving EPP Request 60ms
• From drop notification to send EPP Request 0.0002ms

Receiving the response back from the EPP is slow. I can only imagine their server is a) under load and b) their XML parsing and writing is probably not optimised. There is also a delay on the EPP communication due to the secure socket layer, so encryption is important here.

Servers – Good Network

The golden 2ms from Nominet. I have so far found it hard to be that slow, my DAC send and receive is 350 ticks. I have read this a lot, services boasting about being close to Nominet. Based on my tests, some network monitoring and the above timings Nominet’s servers and coding is your barrier. That is not to say you shouldn’t aim to buy or rent a server on a good network.

Servers – High Spec

You don’t need to high spec machine. Depending on how it is written, your code will probably only run 4 to 6 threads. I’m running a Windows 2008 Quad Core server with 2GB ram and an 80Gb hard drive. Any more would be a waste.

Servers – Only Run Essentials

Only run services that are required, turn everything else off. You don’t want your operating system starting threads the take priority over yours.

Optimising your code and keep track

Keep tweaking your code and never settle for working. I spent one hour rewriting the EPP socket and managed half communication time. Compress data. Remove variables and method calls that are not necessary. Keep yourself connected. Log your transactions, code executions but more importantly your exceptions. Recover from failure; no one likes a site down SMS at 3am! Remember to turn your logging off when live.

Finally are you feeling lucky?

If you base your calculations someone’s system EPP request send taking 5ms (which is slow but possible) you are only as good as your last loop. If your scan intervals are every 200ms and the domain hasn’t dropped you have 199ms before the next check. That’s a long time for other systems to have checked before that next check. Too long! Even at the maximum 60ms intervals, you’ll probably still miss the domain if it drops between scans. It’s all about luck, you have to hope your system pings just at the right time.

I don’t have time tonight but I could also talk about strategy - 60ms, 200ms, multi-checks, blasting, etc. Probably open that one for discussion.

 

[brum]

What a great post good luck gareth and all the best hope u get it all working soon

 

[stevebrowne]

good post I should dust off my catching stuff again.

 

[dotNetFreak]

Very interesting post - Thanks for sharing!

The golden 2ms from Nominet. I have so far found it hard to be that slow, my DAC send and receive is 350 ticks.

Do you mean:

Start Timer -> Send Query -> Read Response -> End Timer

= 0.00350ms?

If so, sorry, but I think your StopWatch class is broken

 

[gareth_brown]

Nothing would surprise me when working with MS products. No, you are right it would be impossible for it to send and receive that fast. The timings are based on the time the data is started to be received. As mentioned above, the check at Nominet can take a few milliseconds depending on the time of day.

I also found a really interesting trick that speeds up Nominet's DAC by about 20% but couldn't end up using it. I wasn't sure if it was a fluke, so I ran two connections and tested 200,000 times on each.

 

[monaghan]

The basics are easy - just simple network port IO, but as you say, the timings are the fun part and how you address the issue of when to poll and at what frequency. From experience, just when you think you have a pattern to work with, the next drop blows away that pattern.

Your server seems very high spec for the task, perhaps try porting to a Linux system as you can generally build a custom kernel with much more control over what you can switch off. The code is going to be in an idle "wait state" for most of the time. Don't forget a decent firewall as those nasty little hackers trying to break into your box will take valuable process time.

Where are you getting your "golden 2ms from Nominet" figure from? I'm seeing more than 2ms between some hops inside Nominet so I presume you are referring to time to the border router?

 

[gareth_brown]

The "golden 2ms from Nominet" are figures quoted by drop catch services. Because no one talks about their scripts its difficult to know what to expect with regards to timings. My point at the bottom is, it doesn't really matter. You either get there before someone else or you don't, with so many milliseconds of gap between each check 'how fast is mine' really doesn't matter. Of course if it takes a 1 second, build the create command, connect to the epp server and send the request you could be struggling!

Linux isn't my bag and for something that is currently a little personal project i'm not going to worry if i'm a millisecond or two slower. I am sure with 356 days in a year I should get lucky one day

 

[monaghan]

Luck is a big part of the process

My first catch system was in C# and sat much further from Nominet than I am now it used the Automaton (long before EPP was available) and it did all right for a 1st bash.

Keep tweaking and tweaking and you'll get there, I've had 2 LLL.co.uk's so far (not with the C# code though) so it is possible to beat "the big boys" if you keep at it. There are a lot more catchers today than when I built my first system, so you need to find something that the others haven't got.

 

[AssetDomains]

Thanks gareth great post gona look into geting my own tag sometime this year. how much would you say you've spent so far geting set up

 

[GlasgowBob]

Great post Gareth, many thanks for sharing your experience getting set up.

I am considering getting my own tag as well this year. Would you say the administrative process with nominet is reasonably straight forward?

 

[monaghan]

Setup costs are server rental (or purchase & co-location fees, although I'd suggest rental at least initially), Nominet membership & DAC fee and whatever you value your programming time at (obviously if doing as a hobby, then this is free).

If you are coding yourself, then there are many GPL tools to help you on your way, if you are coding on Windows, the Visual Studio Express tools will get you on your way for a command line (or GUI) tool, you may struggle with the Express versions if you want to create a Windows service which would be the better way to survive server reboots. You could construct the service code manually, but the VS Professional templates make it much easier!

The Nominet admin side of things is quite straightforward and they are usually very helpful if you have any issues. Even if you don't catch, Nominet membership is a good investment as you can remove your dependency on 3rd party registrar systems.

 

[anthony]

Such a rare type of thread on Acorn these days, it's worth subscribing to! One for freelancerchina.com I think though!

 

[Retired_member41]

There are dozens of hosting companies out there that offer traces to the dac and epp from as little as a third of a ms.

Will this help though lol

 

[gareth_brown]

@newpdc - Money wise I could not tell how much. I would have thought I have spent an excess of 120 hours, which at my hourly rate would be £7,800+. I have been coded various different catching strategies and this is where the time is spent. I wouldn’t what to guess how much it would cost to hire someone to develop it for you. All I do know is unless they have done it before; they will NOT get it right first time. And they will not put the same kind of energy into the project as yourself.

@GlasgowBob – it is a straight forward as any other system.

@anthony – Based on my above comment, I don’t think MR Freelancer is going to give you the time or quality of work you will require. The basic DAC and EPP coding can be done with the TcpClient, which is quick to program but is slower than sockets.

@Caz – Getting hosting companies to trace route is a good selection process. I have so far rented 3 servers at 3 different data centres to test timings.

BUT back to my original post its 99% luck. That is unless you have some at Nominet giving you the nod or you use the DAC command ‘drop,domainname.co.uk\r\n’.

 

[chrisvaughanuk]

BUT back to my original post its 99% luck. That is unless you have some at Nominet giving you the nod

Couldn't agree more, once you've optimised to a certain level, but going back to minimising disruption by other services, judging by the level of technical detail that you go into, I don't think you'll have a problem with Linux!

My script is written in PHP and optimised in much the same way that yours has been. I spent hundreds of hours doing that because I enjoy it and not for the financial reward (though that helps sometimes!). I've often toyed with rewriting in C++ or similar but I am quite happy with the results I'm getting. Plus, my girlfriend has whinged enough at the hundreds or so hours I've already spent! Like you said though, a lot of it boils down to luck anyway, so why waste the time, once you have a nicely optimised system?

Great post Gareth!

Chris.

 

[monaghan]

The trick is to make a start on a re-write, each time I do my "luck" on the current seems to increase, when I decide not to bother rewriting the catch rate falls again I'm sure there's a universal law somewhere that can explain it!

 

[gareth_brown]

PHP isn't my bag but I am sure it won't take long to learn. Have you used something like hiphop to compile your code? I know I'm spending to much time, when my wife starts talking about 'dropping' or asking if i'm optimising JIT again

 

[gareth_brown]

Alex - may be there is...but I am sure there is only so much you can do. Running the system with no expectations has helped me track timings. If you don't have the data, it is too easy to think there's something wrong with your code or server.

On a side note, I did find a way to get 864,000 DAC requests for one month. Although this really messed me up, as I thought I'd stumbled on something no one else was using. As soon as I questioned Nominet about it they stopped me in my tracks. IDIOT!

 

[davedevelopment]

PHP isn't my bag but I am sure it won't take long to learn. Have you used something like hiphop to compile your code? I know I'm spending to much time, when my wife starts talking about 'dropping' or asking if i'm optimising JIT again

I had a very brief play with hiphop when they released the first version, but I use PHP's SSL transport rather than my own and I don't recall it playing nicely off the bat and I didn't have time to look into it.

 

[gareth_brown]

Dave have you tried compiling your php? I am right in thinking PHP is interpreted so by compiling it you could speed your code by at least 30%.