Drop List WHOIS Query DAC

[domainmanuk]

Im after some help please:

We have a DB of 105 million unique 'names', 'terms' pulled from zone files and various scraping all with the extensions removed. We plan to add .co.uk to the full list

Hello

My question is:

We can use the DAC system to query and get whois info for 432,000 per day. But still this would take a very long time, and as you would agree the majority of the 105 million will not be registered with a .co.uk extensions.

Is there a way (a very fast way) to enter this 105 million domains in a system and pull back a simple Y or N to see of they are registered or not ?

Then if they are registered, that DB can be queried with DAC and data saved.

Any help would be appreciated to try and find the fastest way to get results for all of these domains.

Many Thanks,

 

[monaghan]

The only way I can think of is to buy people's non-used DAC allocations to get more queries per day. Anything else is either abusing the public whois system via multiple proxies or buying a drop list to "tick off" existing names. The big problem is the DAC T&C's which prevent more than 1000 names being released to a 3rd party per day.

Either way, you have a lot of work ahead of you

 

[domainmanuk]

so, are you saying that if we have a DAC account and wanted to query the DAC 432,000 times per day from our account for the 105 million domain names, we could not even do that ? I realise this would still take roughly 200 days - and it is the longest way round, but...

 

[Brassneck]

so, are you saying that if we have a DAC account and wanted to query the DAC 432,000 times per day from our account for the 105 million domain names, we could not even do that ? I realise this would still take roughly 200 days - and it is the longest way round, but...

There are not a 105 million .uk domains registered. It is more like under 10 million (from memory).

Any harvesting of the whois records etc is against the Nominet T & C.

Stephen.

 

[monaghan]

so, are you saying that if we have a DAC account and wanted to query the DAC 432,000 times per day from our account for the 105 million domain names, we could not even do that ? I realise this would still take roughly 200 days - and it is the longest way round, but...

I'm sure there are legal and dubious ways of doing what you want to, you'd need to read the T&C's carefully and determine how you apply in each case and see where you fall.

 

[Edwin]

There's absolutely no point at all doing what you're suggesting.

If you run 100 million records against Nominet's 10 million name database, you'll be left with 90 million (or more) unregistered names.

Assuming that you can read through and analyse the value of one unregistered name per second, and that you can sustain that pace 8 hours a day 7 days a week (all incredibly optimistic assumptions) it will take you 3,125 days to check all 90 million unregistered names, or about 8.5 years!

And that's quite aside from the fact that it's A) against Nominet's TOS and B) going to take weeks and weeks of DAC time

 

[domainmanuk]

Many Thanks Again.

Ok, here we go - ive managed a way to get a list from the 100 million of the registered .co.uk domains - the full 100 million will be scanned, and a Y or N will be returned.

This can be done in a week.

So i will then have approx 10 million of registered domains that i need to query against whois.

So - my question is if i use DAC. can i Query the list of 10 million domains with DAC real time, 432,000 per day with no issues - as i have been drawn to this part of the T&Cs -

5.3.6. knowingly provide more than an insubstantial part of the register or a sub-database of it (e.g. a database containing information released under the whois or the DAC) to any one third party or end user in any 24 hour period. For the avoidance of doubt and as an example, more than 1000 registered domain names taken from the DAC would be more than an insubstantial part;


What does this actually translate to for someone in my position ?

Thanks

 

[anthony]

To just query 100 million combinations is pointless surely? Why don't you try prioritising your list first, develop your own strategy of weighting your data via Googles' traffic tools etc, then submit a more focussed and condensed list. It's something you'd need to do anyway with the availables you discover, especially if all you're trying to do is a bit of prospecting for missed nuggets!

 

[retired_member13]

It means you can't pass on lists of more than 1000 names to any 3rd party or Nom can rescind your membership should they choose to (although it appears that in the most part they just warn people).

I'm not sure how you're going to get the 100million done legally in a week though, as public whois has a 1000 per day per IP limit and using multiple proxies is against Nom's T&Cs.

 

[domainmanuk]

Thanks for the reply.

So just to confirm,

I myself, for my own use, can query DAC real time 432,000 per day and get whois info (or some of it) for 432,000 .uk domain names ?

Thanks

 

[aZooZa]

There seems to be mass confusion here regarding DAC and WHOIS output. You can forget trying to obtain 'whois' data on that scale. From the 100 million domains you have or whatever it is, expect to see about 1% return for trying to match them to .co.uk. There are already 2.7 million leaked in general circulation (eons ago) and no doubt most of the the 1 million you find will already be in that file.