mongodb - Best data store w/full text search for lots of small documents? (e.g. a Splunk-like system) -

-

we specing out system index , store zillions of syslog messages. these text messages, few attributes (system name, date/time, message type, message body), typically 100 1500 bytes each.

we generate 2 10 gb of these messages per day, , need retain @ least 30 days of them.

the splunk system has great indexing , document compression system.

what use?

i thought of mongodb, seems inappropriate documents of small size.

sql server possibility, seems perhaps not super efficient purpose.

text files lucene? -- windows file system doesn't dirs zillions of files

suggestions ?

thanks!

i thought of mongodb, seems inappropriate documents of small size

there's company called boxed ice builds server monitoring system using mongodb. argue it's appropriate.

these text messages, few attributes (system name, date/time, message type, message body), typically 100 1500 bytes each.

from mongodb perspective, storing lots of small documents few attributes. in case mongodb has several benefits here.

  1. it can handle changing attributes seamlessly.
  2. it flexibly handle different types.

we generate 2 10 gb of these messages per day, , need retain @ least 30 days of them.

this within type of data range mongodb can handle. there several different methods of handling 30 day retention periods. these depend on reporting needs. poke around on the groups ideas here.

based on people i've worked with, type of insert-heavy logging 1 of places mongo tends fit.


Comments

Post a Comment

Popular posts from this blog

linux - Mailx and Gmail nss config dir -

-
i trying send mail linux command line using mailx command. can send local domain no problem want set mail send gmail account receive mail sent gmail account. after configuring mail.rc so: account gmail { set smtp=smtps://smtp.gmail.com:587 set smtp-auth=login set smtp-auth-user=username@gmail.com set smtp-auth-password=password set ssl-verify=ignore } i error: resolving host smtp.gmail.com . . . done. connecting 74.125.25.109 . . . connected. missing "nss-config-dir" variable. "/home/username/dead.letter" 11/354 . . . message not sent. after looking "nss-config-dir" here , located certn.db , keyn.db files , added mail.rc so: account gmail { set smtp=smtps://smtp.gmail.com:587 set smtp-auth=login set smtp-auth-user=username@gmail.com set smtp-auth-password=password set ssl-verify=ignore set nss-config-dir=/home/user/.mozilla/firefox/location.default } now when try send mail using command: echo "sent gmail account" | mailx -v -a
Read more

c# - Is it possible to remove an existing registration from Autofac container builder? -

-
something along lines: builder.registertype<mytype>().as<itype>(); builder.registertype<mytype2>().as<itype>(); builder.deregistertype<mytype>().as<itype>() var container = builder.build(); var types = container.resolve<ienumerable<itype>>(); assert.istrue(types.count == 1); assert.istrue(types[0].gettype == typeof(mytype2)); scenario: go through bunch of assemblies , go register types want make sure have 1 implementation of given type. need before create container. track on own nice if autofac me bit. this cannot done directly using containerbuilder , unless start on new one. mind you, having first built container should able construct new container filtering away unwanted types , reusing registrations first container. this: ... var container = builder.build(); builder = new containerbuilder(); var components = container.componentregistry.registrations .where(cr => cr.activator.limittype != typ
Read more

php - Mysql PK and FK char(36) vs int(10) -

-
i have read mysql can handle indexes , searches better if tables keys integers rather char(36) uuid's. is worth converting int(10)? note: use 1 server , have have plans use multiple databases concurrently therefore removing 1 of big reasons using uuid. our tables innodb if makes difference. thanks in advance. usually rdbms can handle integer keys pk more efficient, other datatypes. reason how build index column, yes: long dont require string (or other typed) keys, should use integers. however: char(36) , int(10) far away being equal, because int(10) much smaller, char(36). dont know, if require many different keys, should keep in mind. update, complete last paragraph: int(10) 32-bit, char(36) 36 - ^byte^ (= 288-bit). not mean, int consumes less space, means, char(36) has 4 times more different keys.
Read more