php - Analysing alogithm possibly based on regular intervals to check for bots and spiders -

-

i'm trying build script shows me list of ip's bots/spiders.

i wrote script imports access log of apache mysql db can try manage php , mysql.

i've noticed lot of bots have regular intervals, send out request every 2 or 3 seconds. there easy way of showing these patterns query or php script? or, harder think, there algorithm can recognise these bots / spiders.

db:

create table if not exists `access_log` (   `ip` varchar(16) not null,   `datetime` datetime not null,   `method` varchar(255) not null,   `status` varchar(255) not null,   `referrer` varchar(255) not null,   `agent` varchar(255) not null,   `site` smallint(6) not null );

official bots identify themselves. there's list @ http://www.robotstxt.org/db.html

for unofficial ones guess try looking of following:

  • page requests no other resource requests (images, css , javascript etc)
  • strange url requests (lot's of requests login pages, ones don't exist such wp-admin on drupal site)
  • successive page view's in short amount of time
  • exactly same url signatures coming many different ip's
  • no http referrer ip's you've never seen before
  • lot's of comment posts in short session
  • requests public proxy servers

that's of thing's i've noticed annoying ba***s keep trying scrape , spam site anyway. of them need combined in order filter out real requests same characteristics.


Comments

Post a Comment

Popular posts from this blog

Javascript line number mapping -

-
magically formatted comments change reported line number of javascript errors in browsers; this: //@line n "f" n line number , f file name. unfortunately, //@line appears ungoogleable. know there documentation on feature, , browsers support it? (i found references here , here .) from can find available in mozilla based browsers. it using js_setoptions command can turn these options on or off depending on values passed. https://developer.mozilla.org/en/js_setoptions
Read more

c# - Is it possible to remove an existing registration from Autofac container builder? -

-
something along lines: builder.registertype<mytype>().as<itype>(); builder.registertype<mytype2>().as<itype>(); builder.deregistertype<mytype>().as<itype>() var container = builder.build(); var types = container.resolve<ienumerable<itype>>(); assert.istrue(types.count == 1); assert.istrue(types[0].gettype == typeof(mytype2)); scenario: go through bunch of assemblies , go register types want make sure have 1 implementation of given type. need before create container. track on own nice if autofac me bit. this cannot done directly using containerbuilder , unless start on new one. mind you, having first built container should able construct new container filtering away unwanted types , reusing registrations first container. this: ... var container = builder.build(); builder = new containerbuilder(); var components = container.componentregistry.registrations .where(cr => cr.activator.limittype != typ...
Read more

php - Mysql PK and FK char(36) vs int(10) -

-
i have read mysql can handle indexes , searches better if tables keys integers rather char(36) uuid's. is worth converting int(10)? note: use 1 server , have have plans use multiple databases concurrently therefore removing 1 of big reasons using uuid. our tables innodb if makes difference. thanks in advance. usually rdbms can handle integer keys pk more efficient, other datatypes. reason how build index column, yes: long dont require string (or other typed) keys, should use integers. however: char(36) , int(10) far away being equal, because int(10) much smaller, char(36). dont know, if require many different keys, should keep in mind. update, complete last paragraph: int(10) 32-bit, char(36) 36 - ^byte^ (= 288-bit). not mean, int consumes less space, means, char(36) has 4 times more different keys.
Read more