sql server - How to reducing impact on OLTP when performing ETL Process -

-

i new in designing etl process. have 2 database, 1 live database application use every day transaction. other 1 data warehouse.

i have table in live database regularly have new data insert it. goal every night etl process transfer data in live database data warehouse, follow deleting data in live database.

due lack of knowledge, solution got implement call rolling table. on live database, have 2 tables have same structure. call them tbllive1 , tbllive2. has synonym call tbllive. insert done on synonym. synonym point @ 1 of table.

when run etl process, have stored procedure drop , create new synonym point tbllive2. allow etl process transform data tbllive1 without effecting application. assumption etl process takes hour run, , won't want etl process lock table preventing application insert new data it.

this solution should theoretically work, not elegant.

i sure problem common problem, there other solutions out there?

to add bob's answer (above), usual in dwh/bi applications, necessary tables copied "staging" database or "staging" schema on dwh database(depending on number of tables / size etc). these ordinarily on different server oltp system - dwh implementation of size is)

to answer question on performance impact, depends on server spec/io configuration.

is data being inserted oltp system 24hours/day? or there downtimes? or low traffic times?

it might worthwhile using database compression io going biggest enemy , considerably.


Comments

Post a Comment

Popular posts from this blog

linux - Mailx and Gmail nss config dir -

-
i trying send mail linux command line using mailx command. can send local domain no problem want set mail send gmail account receive mail sent gmail account. after configuring mail.rc so: account gmail { set smtp=smtps://smtp.gmail.com:587 set smtp-auth=login set smtp-auth-user=username@gmail.com set smtp-auth-password=password set ssl-verify=ignore } i error: resolving host smtp.gmail.com . . . done. connecting 74.125.25.109 . . . connected. missing "nss-config-dir" variable. "/home/username/dead.letter" 11/354 . . . message not sent. after looking "nss-config-dir" here , located certn.db , keyn.db files , added mail.rc so: account gmail { set smtp=smtps://smtp.gmail.com:587 set smtp-auth=login set smtp-auth-user=username@gmail.com set smtp-auth-password=password set ssl-verify=ignore set nss-config-dir=/home/user/.mozilla/firefox/location.default } now when try send mail using command: echo "sent gmail account" | mailx -v -a
Read more

c# - Is it possible to remove an existing registration from Autofac container builder? -

-
something along lines: builder.registertype<mytype>().as<itype>(); builder.registertype<mytype2>().as<itype>(); builder.deregistertype<mytype>().as<itype>() var container = builder.build(); var types = container.resolve<ienumerable<itype>>(); assert.istrue(types.count == 1); assert.istrue(types[0].gettype == typeof(mytype2)); scenario: go through bunch of assemblies , go register types want make sure have 1 implementation of given type. need before create container. track on own nice if autofac me bit. this cannot done directly using containerbuilder , unless start on new one. mind you, having first built container should able construct new container filtering away unwanted types , reusing registrations first container. this: ... var container = builder.build(); builder = new containerbuilder(); var components = container.componentregistry.registrations .where(cr => cr.activator.limittype != typ
Read more

php - Mysql PK and FK char(36) vs int(10) -

-
i have read mysql can handle indexes , searches better if tables keys integers rather char(36) uuid's. is worth converting int(10)? note: use 1 server , have have plans use multiple databases concurrently therefore removing 1 of big reasons using uuid. our tables innodb if makes difference. thanks in advance. usually rdbms can handle integer keys pk more efficient, other datatypes. reason how build index column, yes: long dont require string (or other typed) keys, should use integers. however: char(36) , int(10) far away being equal, because int(10) much smaller, char(36). dont know, if require many different keys, should keep in mind. update, complete last paragraph: int(10) 32-bit, char(36) 36 - ^byte^ (= 288-bit). not mean, int consumes less space, means, char(36) has 4 times more different keys.
Read more