mongodb - Best data store w/full text search for lots of small documents? (e.g. a Splunk-like system) -

we specing out system index , store zillions of syslog messages. these text messages, few attributes (system name, date/time, message type, message body), typically 100 1500 bytes each.

we generate 2 10 gb of these messages per day, , need retain @ least 30 days of them.

the splunk system has great indexing , document compression system.

what use?

i thought of mongodb, seems inappropriate documents of small size.

sql server possibility, seems perhaps not super efficient purpose.

text files lucene? -- windows file system doesn't dirs zillions of files

suggestions ?

thanks!

i thought of mongodb, seems inappropriate documents of small size

there's company called boxed ice builds server monitoring system using mongodb. argue it's appropriate.

these text messages, few attributes (system name, date/time, message type, message body), typically 100 1500 bytes each.

from mongodb perspective, storing lots of small documents few attributes. in case mongodb has several benefits here.

1. it can handle changing attributes seamlessly.
2. it flexibly handle different types.

we generate 2 10 gb of these messages per day, , need retain @ least 30 days of them.

this within type of data range mongodb can handle. there several different methods of handling 30 day retention periods. these depend on reporting needs. poke around on the groups ideas here.

based on people i've worked with, type of insert-heavy logging 1 of places mongo tends fit.